May 13, 2020·4 min read·engineeringrqliteraftoperations

rqlite adds cluster member removal to its CLI

rqlite 5.3.0 exposes node removal in the interactive client. The command sends a membership request using a Raft node ID and follows leader redirection.

Stopping a database process and removing its membership record are different operations. A cluster can retain a record for a node that will never return, continuing to treat it as a member to contact. rqlite 5.3.0 adds node removal to its command-line client, bringing that management operation into the same interface used for interactive database work. The May 13 release announcement identifies this as the release's new capability.

The server already has a removal endpoint, so the feature is a client integration. Its significance is the path from an operator's node identifier to a request the cluster can process. The implementation also handles leader redirects and configured authentication. Those details matter because cluster management cannot depend on the operator always connecting directly to the current leader.

I value rqlite's compact operational model. Exposing an existing membership operation through its client reduces the number of separate request formats an operator needs to remember. The command still needs to make the target identity and the consequences of membership change clear.

The argument is a Raft node ID

The CLI command dispatcher at this tag lists .remove with a Raft ID argument. It dispatches that command to the removal helper. The identifier refers to the cluster member, not to a SQL row or a filesystem path.

An illustrative invocation inside the interactive client uses an explicitly chosen member ID. This example describes the command and is not a recommendation to remove an arbitrary member.

.remove reader-1

The distinction between identity and address is useful. A node's HTTP address tells the client where to send the request. The ID in the command tells the cluster which membership record to remove. These can refer to different nodes, since the contacted server can process a request concerning another member.

The removal helper creates a JSON object containing that ID and sends an HTTP DELETE request to the configured removal path. Its request target uses the client's protocol, host, port and prefix. This management operation therefore follows the same selected server location as the rest of that client session.

The request must reach the leader

The helper handles a permanent-redirect response by reading the supplied location and issuing the request there. It counts redirects and returns an error if the maximum is exceeded. This is explicit client behavior, which matters for a DELETE request carrying a body. The implementation rebuilds the request from the encoded ID while moving to the indicated leader location.

The server removal handler explains why the redirect can occur. It asks the store to remove the ID. If the store reports that the node is not leader, the handler looks up the leader's API address. It redirects when that address is available and reports service unavailability when it cannot supply one.

The client therefore cannot promise that contacting any reachable process always completes removal. A leader must be available to perform the operation, and the redirect path must lead to it. Reachability of the first HTTP endpoint is only the start of that interaction. The helper reports non-success statuses instead of treating a completed network exchange as a completed membership change.

Authentication accompanies the management request

When credentials are configured, the helper splits the username and password representation and sets HTTP Basic Auth on the request. It rejects an invalid credential format. An unauthorized response becomes an explicit error. This puts authentication handling in the client operation instead of requiring the operator to compose a separate raw request.

The server checks the remove permission before processing the DELETE. Its handler also validates the JSON body, requiring the ID entry. A request can consequently fail before reaching Raft because of authorization or input shape. Those are different failures from the store discovering that the contacted node is not leader.

I prefer that distinction to a management command whose only observable result is whether the HTTP connection succeeded. The operator needs to know whether the server accepted the operation and whether the requested membership change completed. Authentication and leadership are part of reaching that result.

Removal changes membership through Raft

The store implementation first requires local leadership and then calls Raft's RemoveServer with the supplied ID. It waits for that operation's result before proceeding to metadata deletion. This establishes that .remove invokes cluster membership machinery, rather than editing the client's own list of known addresses.

The client helper returns success only for an HTTP OK response after its redirect handling. It does not shut down the removed host, erase its data directory or provision a replacement process. Those are separate operational actions. The new command supplies a direct interface to the membership request, with the server and Raft responsible for interpreting the member ID and applying the cluster change.