rqlite routes node removal requests to the leader

An administration request can reach a healthy database node and still arrive at the wrong place. In rqlite, cluster membership belongs to the Raft leader. A follower can serve an HTTP endpoint without having authority to remove another member, so the management API needs to explain where the request should go.

The rqlite 4.2.3 release fixes that case by returning a redirect when a follower receives a node-removal request. Membership changes remain leader-only. The patch changes how the HTTP service communicates that requirement to a client.

I like rqlite's proportionate approach to putting SQLite behind consensus. A small operational interface is useful only if it handles the topology the system actually has. Following the leader for a membership change is one of those requirements that a distributed database cannot remove by having a simple deployment.

The store already rejects follower removal

The store implementation at the release tag checks whether its Raft instance is the leader before removing a peer. If it is not, the method returns ErrNotLeader. If it is, it calls Raft's RemovePeer and waits for the operation's result.

There is also a second opportunity to receive the not-leader error. Leadership can change after the initial state check but before the Raft operation finishes. The method translates that result into the same store-level error. Checking the role once does not make leadership permanent for the rest of the request.

That behavior preserves the authority boundary. The store does not let an HTTP caller persuade a follower to make an independent membership decision. The new release instead lets the service turn the store's role information into an actionable response.

This division is useful for understanding what the patch guarantees. A redirect is a routing answer. Successful completion of the membership operation is still a separate result that depends on the node receiving the repeated request having the required authority.

The HTTP handler preserves the request's purpose

The removal handler in 4.2.3 accepts DELETE and parses a JSON body containing an addr property. It rejects a malformed body and a body that does not have exactly the expected single field. The address identifies the member to remove.

The handler first checks the caller's removal permission. That is independent of its later discovery that the local node is a follower. An unauthorized request does not gain a routing response that substitutes for authorization.

When the store returns ErrNotLeader, the handler looks up the leader's HTTP address. If it cannot obtain one, it returns service unavailable. Otherwise it constructs a redirect and sends HTTP 301, using Go's StatusMovedPermanently constant.

The redirect constructor retains the request path and raw query string while changing the destination host. This helps the client repeat the same operation at the advertised leader endpoint. The body remains the client's responsibility when it follows or handles the response.

A DELETE carrying JSON deserves explicit treatment in a management client. Receiving a redirect does not mean the removal has occurred. A client needs to send the intended method and address body to the new destination, then examine that operation's result.

HTTP libraries differ in how they handle redirects and request bodies. The important fact for this release is the actual response code and the API's body requirement. A wrapper should make its handling of those properties visible instead of treating every redirect as equivalent to a completed database operation.

A missing leader remains a real failure

The service-unavailable path is as informative as the redirect path. A follower cannot provide a useful destination when it does not know the leader's HTTP address. Returning an error preserves that uncertainty instead of inventing a route.

A membership client consequently has several distinct outcomes to interpret. It can encounter an authorization failure, malformed request data or a node that cannot currently direct the operation. Those failures concern different parts of the management path.

The code also makes clear that the advertised HTTP destination differs from the address passed to the membership operation. The store handles Raft membership, while the service directs an HTTP caller to another API endpoint. Confusing those address roles would make a redirect unusable even if the cluster correctly knew its leader.

This is one reason administrative APIs need their own operational attention. Application data requests and cluster changes may enter through the same service but do not have identical authority or failure conditions. A successful connection to a follower establishes connectivity, not permission to decide membership.

Small release changes improve diagnosis too

The versioned changelog also records logging the store directory path at startup. That is a modest addition, but it identifies which persistent state a process is actually using. An operator diagnosing a management request needs to know both which node answered and which local state that node opened.

The changelog dates 4.2.3 to 21 February, while the GitHub release publication is on 22 February. This post follows the public release date. The routing behavior and startup log entry are both present in that released version.

For me, the useful improvement is the connection between a distributed role and an ordinary HTTP response. The store continues to enforce leader authority, while the handler gives a permitted caller a destination when it can. The client still has to complete the removal request at that destination and distinguish a routing response from a successful membership change.