rqlite can remove unreachable nodes after a configured interval

A node that has been deprovisioned can remain in a database cluster's membership long after its process disappears. The leader continues attempting to reach it until someone removes the record. rqlite 7.11.0, released on November 15, adds automatic removal after a configurable period of unreachability. The tagged changelog identifies this alongside associative query responses.

The new removal behavior is optional. The operator chooses whether to enable it and can choose different intervals for voting members and read-only replicas. I value that distinction because a replica serving extra reads has a different role from a voter participating in consensus. A single failure policy for both would conceal the consequences of removing each kind of member.

Membership persists beyond process availability

The cluster-management guide in 7.11.0 explains that removing a failed node deletes its record from the cluster configuration. Without removal, the leader continues trying to communicate with it. A crashed node that will return and a permanently deprovisioned node can therefore appear similar until the operator supplies a policy or intervenes.

The automatic feature defines an unreachable node as one the leader cannot heartbeat with. That is an observation made through the cluster's communication mechanism. It does not establish whether the remote process has stopped permanently, whether its machine has failed, or whether a network interruption is temporary.

The configured interval turns that observation into a membership decision. A short interval treats a brief communication failure as sufficient grounds for removal. A longer interval retains membership through more temporary failures. The guide recommends conservative values, particularly for voters, because ordinary network interruptions should not accidentally become repeated membership changes.

Voting and read-only nodes have separate settings

The launch flags are -raft-reap-node-timeout for voters and -raft-reap-read-only-node-timeout for non-voters. Both default to zero in the configuration source, with help text stating that no reaping takes place when the option is not set. Existing deployments therefore do not silently acquire a new automatic removal interval when they upgrade the database binary.

The versioned guide gives an example with two days for voters and thirty minutes for read-only nodes.

rqlited -node-id 1 \
  -raft-reap-node-timeout=48h \
  -raft-reap-read-only-node-timeout=30m data

These durations are an example from the documentation, not a universal recommendation. Their different values illustrate that the two classes can have separate lifecycle policies. A read replica intended as replaceable capacity may warrant a different retention period from a member whose vote affects the quorum requirement.

The configuration parser also rejects explicitly supplied nonpositive reap durations. That distinguishes leaving the feature disabled by default from supplying an invalid enabling value. The command-line behavior makes the operator's decision to enable a timer deliberate.

The current leader owns the observation period

The guide states that a leader election resets the timeout clock. An interval therefore does not simply measure elapsed wall time since the last moment any node in the cluster observed the missing member. It belongs to the current leader's observation and removal process.

This matters during repeated leadership changes. A missing node can remain recorded longer than one configured interval measured from its initial disappearance, because each new leader starts its own clock. The setting should be understood as the leader's removal threshold under the documented lifecycle, not an absolute deadline promised across every election.

For consistent operation, the guide requires the settings on every voting node that could become leader. Configuring only the current leader makes the behavior depend on which member wins the next election. Settings on read-only nodes are ignored because those nodes do not become leader and perform the removal policy.

This is a familiar deployment consequence of leader-owned automation. The active process performs the work, but every candidate needs the configuration that defines its behavior. Otherwise failover changes policy as well as process identity.

Removing a voter still requires a functioning cluster

The cluster guide explains that ordinary removal requires an operational leader. A cluster that has lost quorum cannot recover merely by asking its remaining node to remove enough absent members. Membership changes still need the functioning consensus system responsible for accepting them.

A three-voter example makes the limitation concrete. With two available voters, the cluster can continue and remove the failed third member. If two voters are unavailable, the remaining member cannot simply use normal removal to redefine itself as the whole cluster. The guide describes separate recovery procedures for permanent quorum loss.

Automatic removal belongs within that same operational model. It automates cleanup of members observed as unreachable by a functioning leader. It is not a replacement for the recovery procedure needed when the cluster cannot elect and sustain a leader at all.

Read-only replicas have different consequences because they do not count toward the voting majority. Removing an unreachable non-voter cleans up replication membership without changing the number of votes required for consensus. That difference explains why the release exposes a separate timeout for them.

Automation expresses a lifecycle policy

The feature is useful when infrastructure routinely creates and removes replicas while the database should eventually discard their stale records. It lets the cluster apply an explicit retention interval instead of relying on every deprovisioning action to complete a separate cleanup request successfully.

The tradeoff is that heartbeat failure has several possible causes. The timer does not diagnose them. It implements the operator's decision about how long an unreachable member should remain recorded before removal. That makes the duration part of the deployment's membership policy, not merely a performance parameter.

I consider the separate defaults and role-specific timers the useful design choices. Existing deployments retain manual removal unless they enable the feature. Deployments that enable it can configure every potential leader consistently and choose how long to retain voters and read replicas. The resulting behavior remains tied to observed reachability and a functioning leader, which makes its limits clear.