Gossip Connect Seeds Again Scylla

Seedless NoSQL: Getting Rid of Seed Nodes in Scylla

By Asias He

Nodes in a Scylla cluster are symmetric, which ways any node in the cluster can serve user read or write requests, and no special roles are assigned to a particular node. For instance, no primary nodes vs. secondary nodes or read nodes vs. write nodes.

This is a nice compages property. Notwithstanding, there is one small-scale matter that breaks the symmetry: that is the seed concept. Seed nodes are nodes that assist the discovery of the cluster and propagation of gossip information. Users must assign a special role to some of the nodes in the cluster to serve as seed nodes.

In add-on to breaking the symmetric architecture property, do seed nodes introduce any real issues? In this blog mail service, we volition walk through the pains of seed nodes and how we get rid of them to make Scylla easier and more robust to operate than ever earlier.

What are the existent pains of the seed nodes?

First, the seed node does not bootstrap. Bootstrap is a process that a new joining node streams data from existing nodes. However, seed nodes skip the bootstrap procedure and stream no information. It is a pop source of confusion for our users since users are surprised to see no data streamed on the new node afterward adding a node.

2nd, the user needs to decide which nodes will be assigned every bit the seed nodes. Scylla recommends iii seed nodes per datacenter (DC) if the DC has more than than 6 nodes, or 2 seed nodes per DC if the DC has less than 6 nodes. If the number of nodes grows, the user needs to update the seed nodes configuration and modify scylla.yaml on all nodes.

Third, it is quite complicated to add a new seed node or replace a dead seed node. When a user wants to add together a node equally a seed node, the node can not exist added into the cluster equally a seed node direct. The correct mode to do this is to add the node as a non-seed node, then promote it equally a seed node. This is because a seed node does not bootstrap which means it does not stream data from existing nodes. When a seed node is dead and the users want to replace it, they can not use the regular replacing node process to replace information technology directly. Users demand to first promote a not-seed node to act as a seed node and update the configuration on all nodes, then perform the replacing node procedure to replace it.

Those special treatments for seed nodes complicate the administration and operation of a Scylla cluster. Scylla needs to advisedly document those differences between seed and no-seed nodes. Users can make mistakes hands even with the documents.

Can we get rid of those seed concepts to simplify things and brand it more robust?

Let'southward beginning take a closer await into what is the special function of seed nodes.

Seed nodes assist in two means:

ane) Define the target nodes to talk within gossip shadow round

But what is a gossip shadow round? Information technology is a routine used for a multifariousness of purposes when a cluster boots upwards:

• to get gossip information from existing nodes before normal gossip service starts
• to become tokens and host IDs of the node to exist replaced in the replacing performance
• to get tokens and status of existing nodes for bootstrap functioning
• to go features known by the cluster to prevent an one-time version of a Scylla node that does not know whatever of those features from joining the cluster

2) Assistance to converge gossip information faster

Seed nodes assistance converge gossip data faster because, in each normal gossip round, a seed node is contacted once per second. As a result, the seed nodes are supposed to have more recent gossip information. Whatsoever nodes communicating with seed nodes will obtain the more recent information. This speeds up the gossip convergence.

How do we get rid of the seed concept?

To get rid of seen nodes entirely, you will have to solve for each of the functions seed nodes currently provide:

1) Configuration changes

• The seeds option

The parameter --seed-provider-parameters seeds= is at present used only one time when a new node joins for the first time. The only purpose is to find the existing nodes.

• The auto-bootstrap option

In Scylla, the --machine-bootstrap option defaults to true and is not nowadays in the scyllal.yaml. This was intentionally done to avert misuse. It is designed to make the bootstrap process faster when initializing a fresh cluster, by skipping the process to stream data from existing nodes.

In contrast, the Scylla AMI sets --machine-bootstrap to false because nearly of the time when AMI nodes first they are forming a fresh cluster. When adding a new AMI node to an existing cluster, users must set the –auto-bootstrap option to true. Information technology is easy to forget setting the --auto-bootstrap option to true for AMIs and end up with a new node without any data streamed, which is annoying.

The new solution is that the --auto-bootstrap option volition now exist ignored so that we have less unsafe options and fewer errors to brand. With this change, all new nodes — both seed and non-seed nodes — must bootstrap when joining the cluster for the first time.

I small-scale exception is that the first node in a fresh cluster is not bootstrapped. This is because it is the commencement node and there are no other nodes to bootstrap from. The node with the smallest IP address is selected as the get-go node automatically, e.g., with seeds = "192.168.0.one,192.168.0.2,192.168.0.3", node 192.168.0.i will be selected as the first node and skips the bootstrap process. And then when starting nodes to form a fresh cluster, always use the aforementioned list of nodes in the seeds config option.

2) Gossip shadow circular target nodes selection

Before this modify, simply the seed nodes communicated within the gossip shadow circular. The shadow round finishes immediately after any of the seed nodes have responded.

Later this alter, if the node is a new node that merely knows the nodes listed in the seed config option, it talks to all the nodes listed in the seed option and waits for each of them to respond to finish the shadow round.

If the node is an existing node, it will talk to all nodes saved in the arrangement.peers, without consulting the seed config option.

three) Gossip normal round target nodes selection

Currently, in each gossip normal round, iii types of nodes are selected to talk with:

• Select ten% of live nodes randomly
• Select a seed node if the seed node is not selected in a higher place
• Select an unreachable node

Afterwards this change, the selection has been inverse to below:

• Select 10% of alive nodes in a random shuffle + group fashion

For example, there are xx nodes in the cluster [n1, n2, .., n20]. The nodes are first existence shuffled randomly. Then they are divided into 10 groups and then that each group has 10% of live nodes. In each round, nodes in 1 of the groups are selected to talk with. When all groups have talked in one case, the nodes are shuffled randomly again and divided into ten groups. This procedure repeats. Using a random shuffle method to select target nodes is also used in other gossip implementations, e.g., SWIM.

This method helps converge gossip information in a more deterministic way so nosotros can drop the selection of i seed node in each gossip round.

• Select an unreachable node

Unreachable node selection is not changed. It is communicated in society to bring back dead nodes into the cluster.

iv) Improved bulletin communication for shadow round

Currently, Scylla reuses the gossip SYN and ACK messages for the shadow round advice. Those messages are ane way and are asynchronous which means the sender will not await for the response from the receiver. Every bit a upshot, at that place are many special cases in gossip bulletin handlers in order to support the shadow round.

After this modify, a new two way and synchronous RPC bulletin has been introduced for communications in the gossip shadow round. This dedicated bulletin makes it much easier to runway which nodes take responded to the gossip shadow round and waits until all the nodes take responded. It also allows requesting the gossip awarding states that the sender is really interested in, which reduces the message size transferred on the network. Thanks to the new RPC message, the special handling of the regular gossip SYN and ACK message can exist eliminated.

In a mixed cluster, the node will fall back to the old shadow circular method in case the new bulletin is non supported for compatibility reasons. Nosotros cannot innovate a gossip feature bit to decide if the new shadow round method can exist used because the gossip service is not even started when we behave a shadow round.

Summary

Getting rid of the seed concept simplifies Scylla cluster configuration and administration, makes Scylla nodes fully symmetric, and prevents unnecessary operation errors.

The work is merged in Scylla master and will be released in the upcoming Scylla 4.3 release.

Farewell seeds and how-do-you-do seedless!

Larn More than Nearly SCYLLA OPEN SOURCE

langtheamong.blogspot.com

Source: https://scylladb.medium.com/seedless-nosql-getting-rid-of-seed-nodes-in-scylla-d403ea3313e7

Share this post