rqlite 10 stops slow followers from blocking leader snapshots

A persistently slow follower could keep an rqlite leader from taking new snapshots. The follower needed a snapshot to catch up, but streaming an existing snapshot to it blocked the leader's next snapshot operation. Retrying normally worked. A follower that kept requiring snapshots could repeat the interference until the leader starved.

The rqlite 10.0.0 release, published April 28, changes that relationship. Its tagged changelog dates the version April 27 and says leader snapshotting no longer blocks on other operations in the snapshot store. Streaming a pre-existing snapshot to a follower is the concrete example.

This is a change to the interaction between replication maintenance and leader progress. It is not an announcement of a new SQL transaction isolation level. The release's significance is that work required to recover a slow member no longer prevents the leader from producing a new snapshot through the same store.

I like rqlite's proportionality. It supplies distributed relational state through SQLite and Raft without asking the application to operate a larger database platform than the problem needs. Preventing one struggling follower from repeatedly blocking leader maintenance supports that preference in a practical way.

Recovery work no longer excludes new snapshots

The old behavior coupled two operations with different purposes. A transfer sends an existing snapshot to a follower. Creating a new snapshot advances the leader's own snapshot state. The changelog identifies the snapshot store as the place where those activities previously interfered.

That coupling became particularly troublesome when the follower was slow enough to need snapshots repeatedly. A single delay could be followed by a successful retry, but repeated transfers could keep winning against new snapshot attempts. The resulting starvation depended on the persistence of the slow-follower condition.

Version 10 changes the store so that snapshot creation is not blocked by those other operations. The changelog associates the non-blocking store with pull request #2471. That change merged February 17, showing that the implementation predates the April release announcement. The released behavior is documented in the version's own changelog.

The practical benefit follows directly from the reported failure mode. A leader can continue taking snapshots while it transfers an earlier one. The release does not claim that this makes a slow follower fast or removes the work required for recovery. It removes a dependency that allowed that work to stop another maintenance operation.

The wording also has a useful limit. It refers to other operations in the snapshot store. It does not promise that no part of snapshot creation ever waits on any resource. The same release separately describes interference from long-running database reads, which requires a different response.

Long reads cause snapshot retries

Before version 10, a long-running read could block Raft snapshotting for an extended period and, as a result, block writes. The new behavior aborts the snapshot process and retries later when such a read is active. This is a deliberate choice about which operation should yield.

That choice differs from the slow-follower repair. The store now permits work that previously blocked snapshot creation, while the database interaction can cause an attempted snapshot to stop and try again. Both changes reduce unwanted interference, but describing every snapshot as simply non-blocking would hide the retry behavior.

There is an observable tradeoff in the published description. A snapshot attempt may not complete while a long read remains active. That is preferable to letting the attempt hold up writes for an extended period, under the problem statement in the changelog. The release changes how contention is handled rather than claiming contention has ceased to exist.

This is the part of a small distributed database that interests me most. The API may make reads and writes look simple, while background maintenance determines how they interact under less cooperative workloads. A release that names the interference and changes which operation yields gives operators something specific to understand.

Snapshot integrity receives broader checking

Version 10 also uses CRC32 checks more comprehensively to detect filesystem corruption and inadvertent modification of underlying data files. The changelog names both as examples. This is detection of changed bytes, separate from the concurrency changes that govern when snapshot operations proceed.

Several implementation entries explain where the additional checking applies. The snapshot store checks CRC32 values at startup. Snapshot restore checks CRCs, and full-snapshot sinks validate data against checksums in their headers. The release also records checks during reaping and the use of structured sidecar files for stored CRC32 values.

These details make the claim more useful than a general statement that snapshots are safer. They identify points where rqlite checks the files it consumes or maintains. A changed file can be detected when the store opens or when snapshot data passes through validation, according to the listed changes.

Checksums do not repair a damaged database or establish the origin of its contents. The changelog describes corruption detection, and that is the property the release supports. Operators still need backups and recovery procedures. The upgrade section explicitly asks users to back up their system before moving to version 10.

The console is a convenience over the same database

The release adds a built-in web application at /console, available by default at http://localhost:4001/console. It supports working with the deployment, including running queries. This is a useful addition for inspecting relational state without assembling a separate interface.

The versioned README continues to show the HTTP API. A query remains a request against the database endpoint, for example the documented retrieval of rows from a table named foo.

curl -G 'localhost:4001/db/query?pretty' \
  --data-urlencode 'q=SELECT * FROM foo'

The example assumes the node is running locally and the table exists, as in the README's preceding creation step. The console gives another way to interact with that deployment. It does not imply a new storage model or a change to the clustering semantics that the release explicitly says remain compatible.

This combination is consistent with the project I find useful. The release does substantial internal work on snapshot progress while keeping ordinary access approachable. The console is visible to users, but the maintenance behavior is likely to matter more when a cluster member falls behind.

Rolling upgrades have a specific membership restriction

The upgrade instructions say transitions from version 7 or later are supported, with an initial upgrade to the latest version 9 recommended. Version 10 supports rolling upgrades. However, a new version 10 node cannot join a version 9 or earlier cluster.

Those statements fit together only if replacement and upgrade are treated carefully. The documented sequence is to upgrade the existing cluster to version 10 before adding new nodes. Interpreting rolling-upgrade support as permission to start by joining a fresh version 10 member would contradict the explicit membership restriction.

The release has no breaking API changes and says clustering behavior is unchanged, but it does alter command-line options. It removes -on-disk-path, which the changelog says provided little benefit and made node corruption too easy. It renames -raft-timeout to -raft-heartbeat-timeout to describe the option's purpose more accurately.

That rename matters to launch configuration even when application requests remain compatible. A deployment can have a stable HTTP API and still need its service arguments updated. The addition of -raft-commit-timeout is separate, exposing control over another Raft configuration option in this release.

Downgrade also has an explicit procedure. The project says to back up the version 10 system and use that backup to deploy a new pre-version-10 system. It does not offer an in-place reversal by swapping the old binary back into the existing data directory. That restriction belongs in the release's operational account alongside rolling-upgrade support.

The changelog also exposes snapshot-transfer compression as an on/off choice and states that it is off by default in the final feature list. An earlier implementation entry records a different default during development. For the released behavior, the final feature summary is the relevant statement. This is another reason to read the release description as a whole instead of treating every merged change as an independent account of the shipped configuration.

For me, the main improvement remains the separation of snapshot creation from transfer. It addresses a specific way that a slow member could interfere with the leader, while the long-read change prevents snapshot attempts from extending write blockage. The documented upgrade sequence provides a way to obtain those changes without assuming that new-node joins and existing-node upgrades have identical compatibility rules.