rqlite discovers cluster join addresses through a shared registry

Forming a cluster requires an answer to a question that consensus does not answer by itself. A new node must know where to send its first join request. rqlite 3.12.0, released on 1 March, integrates an external Discovery Service so that nodes can learn one another's join addresses without having those addresses fixed in their startup commands.

The release's discovery guide describes the earlier sequence. An operator starts one node, waits for it to become leader and passes its address to later nodes. That is straightforward when addresses are known. It becomes inconvenient when infrastructure assigns addresses at startup.

I value rqlite's effort to keep deployment proportional to the problem. Discovery removes a specific piece of startup coordination without replacing the database's consensus machinery. The useful distinction is between learning an address at which to ask for membership and actually becoming a member of the replicated cluster.

An identifier groups registrations

The operator first creates a Discovery ID. The documentation supplies a POST request to the hosted service at discovery.rqlite.com. Its response contains the identifier, a creation timestamp and a list of registered nodes. The same identifier is then supplied to the nodes intended to form one cluster.

curl -XPOST -L -w '\n' http://discovery.rqlite.com/

This is the historical command documented for the service. Its purpose is to create a registration group, not to create tables or start a database process. The returned identifier is configuration input for the subsequent node startups.

Each node registers an address under that ID and receives the current list of registrations. The first node receives a list containing itself and can form the initial single-node cluster. Later nodes receive addresses they can use as join candidates. The service supplies a rendezvous point for the startup sequence.

The guide shows three nodes starting with the same identifier. Different HTTP and Raft ports in its example let all three run on one host. That arrangement demonstrates the mechanics, while the guide notes that a real deployment normally places the nodes on separate hosts. Discovery does not make several processes on one machine independent machine-failure domains.

The registered address is an API address

By default, a node registers the address supplied with --http. If --httpadv is set, it registers the advertised HTTP address instead. The guide identifies this override explicitly. It is useful when the address used for listening differs from the address that another node must contact.

The 3.12.0 startup code makes that choice before registration. Its determineJoinAddresses() function starts with the HTTP address, replaces it with the advertised address when supplied and registers the result. It excludes its own address from the returned candidates.

The join operation separately supplies the joining node's Raft address. If a Raft advertise address exists, startup uses that value. These two address choices serve different purposes. The HTTP address identifies where the join request goes, while the Raft address tells the cluster how to contact the joining node for its replication traffic.

Confusing them would leave a registry full of addresses that do not name the intended interface. The discovery service is useful only if its returned candidates are reachable as join endpoints. An advertised address is therefore part of the deployment's connectivity model, beyond the fact that a local process successfully opened a listening socket.

Discovery supplies candidates to the existing join path

The same startup function gives an explicit join address first priority in the candidate list. It then adds other addresses returned by discovery. After determining the list, startup opens the store and checks whether joining is required before issuing a join request.

If the node already belongs to a cluster, the startup path logs that it is ignoring join requests. The documentation states this caveat directly. A returned registry list does not override existing local membership simply because startup can obtain it. That separates discovery configuration from the persisted state of a node that has already joined.

The join implementation walks candidate addresses in order. It tries the list up to three times, with a five-second interval after a failed pass. The release therefore combines discovery with retries, which matters when several new processes begin at nearly the same time and their API listeners are not ready in exactly the same order.

This is bounded retry behavior in the join path. It is not an indefinite controller that continuously converges registry entries into cluster membership. The implementation returns after success or after exhausting its attempts. Startup treats an unsuccessful required join as a failure rather than silently declaring that the node has become part of the desired cluster.

I find that distinction helpful because automatic discovery can sound more comprehensive than the code actually is. Version 3.12.0 automates how a node obtains initial contacts and retries joining them. It still requires reachable interfaces, valid configuration and a successful membership operation.

Redirects retain the join request

The join helper resolves the joining Raft address and sends it as JSON to a candidate's /join endpoint. Its HTTP client disables automatic redirect following. When it receives a 301 response, it obtains the next URL from Location and constructs the POST again.

This makes redirection part of the join protocol implemented by rqlite. The helper keeps the joining node's address in the request body when contacting the next endpoint. If a redirect lacks its destination, the helper returns an error. A successful HTTP response identifies the endpoint that accepted the join.

The discovery client uses a similar explicit redirect approach for registration. It posts the registering address to the URL formed from the service location and Discovery ID, then decodes a successful response containing the current nodes. Unexpected response status becomes an error.

These clients perform different jobs despite both using HTTP. Registration describes an address to the registry. Joining asks an existing database member to admit a node. A registry response containing a node is therefore not proof that the node has completed the second operation.

Registry lifetime and cluster lifetime differ

The guide provides a DELETE request for removing a registered address, for example after a node has permanently left a cluster. That operation changes the Discovery Service's list. It should not be confused with removing a node from Raft membership, because the registry and the cluster maintain different information for different reasons.

Stale registry entries affect which endpoints a new node tries. The ordered candidate list and retries can help when another usable endpoint remains, but they do not turn an obsolete address into a reachable node. Removing obsolete registrations keeps future bootstrap attempts directed toward relevant contacts.

The hosted service is offered as a convenience on an as-is basis, without an availability guarantee. The documentation also says that operators can deploy the service source themselves. Choosing the hosted service therefore adds an external dependency to the discovery-based startup path, while self-hosting moves responsibility for that registry to the operator.

This dependency is specific to the workflow being introduced. The service gives nodes addresses for cluster formation. The inspected implementation does not route SQL execution through the registry or ask it to approve Raft log entries. Keeping that role narrow explains how discovery can simplify startup without becoming the database's authority for committed relational state.

The returned node list also does not contain the application database. Its response structure has a creation timestamp, an identifier and address strings. A successful registration establishes that the registry accepted the address and returned its current list. Data synchronization must follow through the actual cluster protocol after joining. This matters when interpreting startup logs, since a registration success and a join success report completion of different operations.

Explicit join configuration remains useful when addresses are already known. Because startup places that contact first, discovery can supplement a supplied contact instead of requiring an entirely separate server executable or SQL interface. Both choices lead into the same join helper.

For me, the attraction is the reduction in address coordination. A shared Discovery ID replaces the need to distribute the first node's assigned HTTP address to each later startup. After registration has supplied candidate contacts, rqlite still performs a join using the node's Raft address and relies on the cluster's existing replication mechanisms for database changes.