rqlite lets removed nodes rejoin at their previous address

A replacement node can occupy the same network address as the node it replaces while carrying a different history. Cluster membership and local persistent state have to agree on which participant that process represents. Treating an address as sufficient evidence of continuity can make a routine replacement difficult to reason about.

The rqlite 4.3.1 release, published on 12 October 2018, changes what happens when a node detects its removal from a cluster. A removed node no longer shuts down its Raft subsystem at that point. The release says this allows rejoining with the same IP address, provided all of the node's local data has first been deleted.

I value rqlite's modest operational requirements, and predictable replacement belongs among them. The qualification about local data is essential. The change enables reuse of an address with a fresh node state, rather than making an old data directory interchangeable with a newly joined member.

Removal is a cluster operation

The cluster-management guide at the release tag distinguishes a process failure from removing a member. A crashed process can ordinarily restart and catch up. Removing a node changes the cluster's record of its participants.

The guide describes removal through the HTTP API, supplying the member's Raft address. The store implementation requires leadership for that operation and passes the address to Raft's peer-removal mechanism. A process being absent from the network is not itself the membership update.

This distinction matters when replacing a host. Stopping a daemon does not necessarily tell the rest of the cluster to stop treating it as a member. Conversely, restarting a daemon that has been removed does not establish that the cluster has admitted it again.

The management workflow therefore has both a remote side and a local side. Membership changes in the cluster, while the process retains or replaces its local state. The release's rejoin condition ties the second operation to the identity represented by the reused address.

Raft shutdown was a separate lifecycle decision

The vendored Raft configuration documents ShutdownOnRemove. Removing the local node makes it forget peers and transition to follower state. The option determines whether Raft also shuts down.

The library's default sets that option to true. rqlite's store builds a Raft configuration and explicitly assigns the store's ShutdownOnRemove field to it. That makes removal behavior a choice made by the application, instead of an unavoidable consequence of the peer-removal operation.

In 4.3.1 the store's ordinary zero-valued Boolean gives the non-shutdown behavior described by the release. The subsystem can remain alive after learning that its node was removed. Staying alive and remaining a member are different facts.

This is a useful distinction in a long-running database process. The HTTP service and Raft subsystem have related but separate lifecycles. A process can still exist while its consensus subsystem has stopped, and observing the process alone does not establish that it can participate in subsequent cluster operations.

The patch addresses that lifecycle impediment. It does not turn removal into a temporary network outage or automatically preserve the removed member's authority. Rejoining remains an explicit operation with the local-state condition stated in the release.

A fresh join is different from restarting existing state

The tagged guide says an existing member does not need another join argument when it restarts. The argument is ignored when the node is already part of a cluster. That behavior makes an ordinary restart convenient, but it also demonstrates that startup depends on the node's existing state.

A replacement prepared with cleared local data follows the new-node path. It needs to contact a cluster and join, rather than relying on an old record of membership. Reusing the former address does not remove that difference.

The guide also explains that a join can contact a follower, which redirects it to the leader. Before adding a peer, the store still checks leadership. The joining process can use an entry address without acquiring authority to choose its own membership.

Deleting local data is consequently a substantive reset. It is not a cosmetic cleanup of an unused cache. The release specifically requires clearing all data beneath the node before this same-address rejoin case, so describing only a restart would omit the operation that makes the stated behavior applicable.

No destructive shell example is necessary to explain that condition. The important information is which node's local state is being replaced and which cluster remains the source of its new state. Confusing the replacement's directory with a surviving member's directory would describe an entirely different operation.

SQLite files do not replace consensus state

The store code also explains its treatment of an on-disk SQLite database. When opening that form of storage, it removes the database file because the state will be rebuilt from a snapshot and committed log entries. The SQLite file is therefore not the only durable information relevant to membership and recovery.

That design fits rqlite's use of replicated statements. A process needs the consensus history and associated state that determine what to apply. Having a familiar relational file at a familiar address does not by itself prove that the process is a current cluster member.

For me, the practical improvement is a more usable lifecycle after removal, with a precise boundary around state reuse. Version 4.3.1 leaves Raft running so the previous shutdown behavior does not prevent the documented replacement case. A same-address rejoin still begins with cleared local data and obtains its place in the cluster through the join mechanism.