January 24, 2021·4 min read·engineeringrqliteraftstartupsnapshots

rqlite 5.9 makes startup waiting an explicit policy

Leader discovery and local log application are separate startup conditions. Version 5.9.0 exposes the leader-wait choice and derives trailing-log retention from the snapshot threshold.

Starting a distributed database process does not establish that it can serve a useful request. It may still need to find a leader or apply existing log entries to its local database. The January 24 release of rqlite 5.9.0 concentrates on this startup behavior. Its new control determines how the node treats failure to find a leader within the opening timeout.

The release also introduces a policy for retaining trailing Raft logs. These changes address different points in the node lifecycle, deciding when startup may continue and determining how much log history remains alongside snapshots. The tagged changelog identifies both changes, along with clearer startup logging and a CLI version display.

I like rqlite's proportionate approach to replicated relational data. SQLite handles the local database while Raft coordinates changes. The startup policy matters because those responsibilities become ready at different times, and an operator needs to know which condition a process is waiting for.

Finding a leader and applying the log are distinct

The startup code in the release tag defines raft-leader-wait, defaulting to true. It also defines raft-open-timeout, defaulting to 120s. The waitForConsensus function first asks the store to wait for a leader using that timeout.

If leader discovery returns an error and leader waiting is enabled, startup returns an error explaining that the leader did not appear within the timeout. When leader waiting is disabled, the function logs that it is ignoring the leader-wait error and continues. Disabling the flag therefore changes the treatment of that failure. It does not create a leader or make consensus available.

rqlited -raft-leader-wait=false node-data

This illustrative command uses the option present in 5.9.0. It should not be interpreted as a guarantee that startup skips every wait. The implementation still invokes the leader wait and decides how to handle its result. Its next stage separately concerns applying existing logs.

When the opening timeout is nonzero, waitForConsensus calls WaitForApplied. An error there still stops startup. When the timeout is zero, the function logs that it will not wait for logs to be applied. These branches make the difference between accepting a missing leader and skipping local replay waiting explicit.

A process supervisor can observe whether a process remains running, but that observation alone does not describe either condition. The flags express which startup failures the executable accepts. Request behavior still depends on what the node can actually do after it starts serving.

Trailing logs follow the snapshot threshold

The store implementation at v5.9.0 deliberately avoids exposing direct control over trailing logs. Its comment states that it implements a policy instead. The code derives the retained count by multiplying the snapshot threshold by 1.25, then converting the result to an unsigned integer.

The default snapshot threshold in the startup code is 8,192 entries. Under the new formula, that yields 10,240 trailing entries. This is a count of log entries, not a fixed disk-space budget. Different commands can occupy different amounts of serialized data, so the formula should not be described as a byte limit.

The store passes its derived count to Raft's TrailingLogs configuration and exposes it as trailing_logs in status. That gives an operator a concrete value to inspect while leaving the relationship to snapshot policy in the implementation. The new behavior is an automatic retention policy, not an independent command-line retention setting.

Snapshots and retained logs concern the representation of replicated history. The store's own type comment describes it as a SQLite database whose changes go through Raft consensus. Retaining a suffix of the log does not change that write model. It changes how much command history remains when snapshotting and compaction manage older state.

I prefer documenting the derivation because it makes the operational consequence of changing the snapshot threshold visible. A threshold adjustment also changes the trailing count under this policy. Treating the two settings as unrelated would give an incomplete explanation of what the executable configures.

Status improvements explain startup work

The changelog also records clearer Raft log status during startup. That is relevant to the new waiting control because a delayed node can be doing different kinds of work. A leader-discovery delay and a local application delay need different explanations, even when both appear to the caller as a process that has not begun answering requests.

The CLI's new version display addresses another identification problem. A cluster can contain processes and client tools built from different revisions. Showing the contacted node's version at startup gives the user direct information about the service involved in the session. It does not itself prove compatibility, but it removes one avoidable ambiguity.

Version 5.9.0 therefore makes startup policy and retained-history policy more specific. The leader flag controls whether one startup error is fatal. The nonzero opening timeout still governs a separate application wait, while trailing-log retention follows the configured snapshot threshold. Those are the actual choices an operator receives in this release.