rqlite 6.6 can rebuild membership after permanent quorum loss

A cluster that has lost its majority cannot ordinarily agree to remove the missing voters. That makes permanent quorum loss different from replacing one failed machine in an otherwise healthy cluster. The September 21 release of rqlite 6.6.0 adds an explicit recovery procedure for that condition. An operator supplies the membership of a recovered cluster, and the remaining nodes reconstruct state through a recovery path at startup.

The new capability does not promise that every previously acknowledged or attempted change survives an arbitrary loss of machines. The tagged cluster-management documentation calls this partial recovery. It warns that missing servers can leave incomplete knowledge of committed state and that recovery implicitly commits outstanding Raft log entries. The procedure can therefore include changes that had not committed before the failure.

I value rqlite's relatively small operational model, but permanent loss of quorum is a point where the operator must make a consequential decision. This release gives that decision a documented mechanism. Its usefulness depends on understanding the authority the recovery configuration replaces and the limits of the surviving data.

Ordinary membership changes need an operational cluster

The tagged management guide explains the majority requirement with concrete cluster sizes. A three-voter cluster needs two voters to remain operational. If one fails, the other two can still agree on changes. If two fail, the remaining node cannot simply decide that it is now the whole cluster through the ordinary removal API.

That restriction prevents membership change from becoming an automatic way for every isolated minority to declare itself authoritative. A node losing contact with peers does not know merely from that observation whether the peers are gone forever or operating elsewhere. Treating a temporary partition as permanent loss would give the recovery decision a different meaning from the one documented here.

The guide states that graceful node removal generally fails in this condition because no operational leader can process it. A recovery procedure therefore cannot be described as just another client request to remove failed nodes. It operates below that request path, using retained local state and an explicitly supplied configuration.

This is also why a smaller new membership can make progress after recovery. The new configuration defines which voters participate. It does not obtain the missing old votes through a hidden network operation. The operator has established which surviving set is intended to continue the database.

The recovery file names the intended members

The documented procedure begins by stopping all remaining nodes. An operator then creates raft/peers.json under each node's data directory. Its JSON array identifies every intended member by node ID, Raft address, and whether it is a non-voter. The file must be identical across all remaining nodes.

[
  {
    "id": "1",
    "address": "10.1.0.1:4002",
    "non_voter": false
  },
  {
    "id": "2",
    "address": "10.1.0.2:4002",
    "non_voter": false
  }
]

This is an illustrative membership file using the documented fields. Its IDs and addresses are placeholders for the intended recovered members. A node's ID must retain its previous value, while its Raft address can change. The file's address is the inter-node address, not an HTTP client URL.

The non-voter field controls participation in voting. Omitting it defaults to false according to the guide, which means ordinary voting membership. The distinction matters because merely listing machines does not establish how many can contribute to the new quorum. A read-only non-voter does not supply an additional vote.

The documentation requires confirmation that omitted nodes have failed and will not later rejoin. That is a substantive prerequisite of assigning a new membership. The recovery file cannot itself establish what another machine is doing outside the surviving set. Its contents express the chosen configuration after that operational fact has been established.

Reconstruction uses snapshots and remaining logs

The store implementation at v6.6.0 provides a detailed account of reconstruction. RecoverNode validates the supplied configuration, then asks the snapshot store for its available snapshots. It attempts to restore them in newest-to-oldest order. If snapshots exist but none can be restored, recovery returns an error.

The code extracts the database bytes from a usable snapshot and records its index and term. It creates a temporary in-memory SQLite database from those bytes, or starts an empty temporary database when no snapshot data is available. That local representation allows it to replay remaining commands and produce a new snapshot.

Next it reads log entries after the restored snapshot through the log store's last index. Command entries go through applyCommand against the temporary database. The loop updates the last index and term as it advances. This is the mechanism behind the warning about outstanding entries, recovery replays retained history through that endpoint rather than consulting a surviving old quorum for each entry's commitment status.

The implementation then creates a new snapshot containing the recovered database and the supplied configuration. After persisting and finalizing that snapshot, it compacts the old log range. A source comment says this prevents interference from configuration-change entries that may remain there.

These steps show why editing the file is more than changing a list used by the networking layer. Recovery reconstructs state and establishes a new snapshot with a new membership configuration. The retained commands, snapshot contents, and supplied configuration jointly determine the resulting node state.

The validator checks structural mistakes

The recovery configuration checker rejects an empty ID or address, duplicate IDs, and duplicate addresses. It also requires at least one voter. These checks catch mistakes that would make the supplied membership internally invalid before the recovery procedure proceeds.

They do not prove that the selected members contain every desired update. A structurally correct JSON configuration can still describe an operationally mistaken survivor set. The missing information after losing machines is exactly why the documentation states a data-loss boundary rather than claiming ordinary consensus guarantees for this procedure.

I find the explicit replay behavior particularly important. Recovering availability can also change which attempted operations become visible in the SQLite database. Application reconciliation needs to consider both missing changes and changes that the failed cluster had not yet reported as committed.

That consequence is specific to the documented procedure. It should not be generalized into a claim that normal Raft operation commits arbitrary outstanding changes. The recovery code is an exceptional path for a cluster that cannot use normal membership and consensus operations to restore service.

Address changes can create the same need

The release announcement and management guide also identify another use case. A cluster can restart with all nodes, or a quorum of nodes, at different Raft IP addresses. The guide says this situation requires the recovery process too. The machines may still have their data while the retained membership addresses no longer connect them correctly.

The stable-ID rule helps explain this case. A recovered member retains its identity but can receive a new address in peers.json. The operation reconnects identity, reachable transport, and intended voting membership. It is different from changing an HTTP address used only by external clients.

Ordinary address replacement in an operational cluster can use rejoining, as the same guide describes. The recovery feature is necessary when the cluster cannot form the quorum needed for that ordinary path. The availability of one mechanism does not make it the preferred description of every node restart or address adjustment.

The trigger becomes a recovery record

After successful recovery, the service renames peers.json to peers.info. The latter does not trigger another recovery and remains as a record for reference. This makes the file's lifecycle explicit, its first name requests the exceptional startup operation, while its second records that the operation completed.

The 6.6.0 changelog lists this recovery capability as the release's new feature. Other diagnostic and readiness additions in the cumulative history belong to preceding releases. The September change here is specifically the ability to continue from surviving state after the old membership can no longer function.

That continuation has an explicit boundary. The operator stops the survivors, supplies the same intended membership to each, and excludes machines that must not return as old members. The implementation restores a usable snapshot, replays retained commands, and writes a new snapshot containing that membership. Its recovery contract includes possible data loss and possible commitment of previously outstanding entries, which are necessary parts of understanding the database it brings back into service.