rqlite 8 snapshots changed pages to support larger databases

A replicated database can reach a practical size limit because of maintenance work that does not appear in an ordinary query. rqlite's earlier snapshot process consumed more memory and time as its SQLite database grew. Taking a snapshot repeatedly brought the size of the entire database into that operation. The 8.0.0 changelog, dated December 5, describes a redesign intended to remove that scaling limit.

The release uses a WAL-based snapshot store. After an initial full snapshot, the running database can supply changed pages from its SQLite write-ahead log. The snapshot store folds those changes into its retained database file. Understanding both halves is important, since the system still needs a complete recovered state even when producing the next snapshot no longer requires serializing the entire live database.

I value rqlite's proportionality, and this redesign improves a part of that model that can otherwise become disproportionate as data grows. A small application request should not imply that every later maintenance operation must copy all historical data through memory. The code makes the improvement concrete while also showing where resource use still depends on workload.

The first snapshot establishes a complete base

The store's snapshot method first lists the snapshots available in its snapshot store. If none exist, it selects a full snapshot. It checkpoints the live SQLite database, opens the database file, and wraps that file as the snapshot reader.

The snapshot wrapper persists its input with io.Copy. For the full case, the input is the database file. This streams the contents into the sink instead of first constructing one byte slice containing the whole database. The database still needs to be read and written in full for that initial base.

That distinction limits the meaning of the release's scaling claim. Establishing a first snapshot necessarily handles the complete database contents in this implementation. The benefit concerns avoiding that same whole-database preparation on every subsequent local snapshot. It is not a claim that the first copy takes constant time regardless of database size.

The snapshot operation also takes the store's query transaction lock while preparing its snapshot input. That places snapshot preparation within the store's synchronization scheme. Replacing whole-database serialization therefore changes a coordinated maintenance operation, rather than merely changing an optional export command.

Subsequent snapshots read the WAL

When a base snapshot exists, the store selects its incremental branch. If the live database has a WAL file, it opens that file and constructs a fast compacting scanner. A WAL writer consumes the scanner and writes its result into an in-memory buffer. The store then checkpoints the live database and creates a snapshot reader over the compacted buffer.

The buffer is an important detail. Incremental snapshot preparation is not free of memory allocation. Its size follows the compacted WAL output produced for that interval, rather than requiring a representation of every page in the database. A workload that changes many pages can still produce a substantial snapshot.

The method records both the original WAL size and compacted output size. Those measurements describe different amounts of work. The original file can contain repeated changes to a page, while the compacted output can retain a smaller set of page versions sufficient for updating the base snapshot.

Checkpointing follows extraction of the relevant WAL contents. That ordering preserves the changes needed for the snapshot before the live database checkpoint processes its WAL. The snapshot store receives the extracted representation through its own persistence interface.

Compaction keeps the final committed page versions

The compacting scanner tracks frames by page number. It accumulates the frames belonging to a transaction and copies their latest entries into the committed-frame map when it reaches a committing frame. A later committed frame for the same page replaces the earlier entry in that map.

After scanning, it sorts the retained frames by their original offsets. Its iterator then seeks to each retained frame and reads the corresponding page data. This produces the latest valid frame for each affected page in the order the writer needs. The scanner rejects a file that ends with an open transaction according to its commit tracking.

A simple conceptual example explains the saving. Suppose one interval changes the same page repeatedly while leaving most database pages untouched. The base snapshot already contains those untouched pages. Retaining every intermediate version of the changed page would preserve work that is unnecessary for producing its final state. Compaction selects the committed version needed by the new snapshot.

This is snapshot construction, not an application-visible history of every SQL operation. Removing superseded page versions is appropriate for representing a database state. A requirement to reconstruct each earlier business event would need a different retained history and access mechanism.

The fast scanner also has an explicit assumption. Its constructor skips the full checksum scan used by the more defensive mode, because the caller treats the WAL as valid and free from leftover pages belonging to a previous checkpoint cycle. That contract matters when understanding why the scanner is used inside the controlled database snapshot path.

The snapshot store merges the incremental data

The snapshot sink writes incoming bytes into a temporary snapshot directory. Finalization inspects the data to determine whether it is a SQLite database file or a SQLite WAL file. A full database becomes the snapshot database file directly.

A WAL requires an existing snapshot database. The sink rejects WAL data for the first snapshot because there would be no base to update. When a base exists, it stores the incoming file under the WAL name associated with the new snapshot, then moves the previous database file into the new snapshot's database path.

Opening and closing that database replays the WAL into it. This produces a complete SQLite database for the new snapshot. The method updates metadata with the resulting database size and reaps old snapshot data. The retained state is therefore usable as a full database even though its latest update arrived as changed pages.

This division of responsibility explains the design's efficiency. The running database supplies incremental data, while the snapshot store owns the complete accumulated snapshot. New snapshot production does not need to reconstruct that complete accumulated state in a single in-memory serialization of the live database.

The snapshot store also limits creation to one active sink through a mutex held until close or cancellation. Its comments explain that sinks depend on the store's current state. Serializing finalization prevents two sinks from simultaneously changing the base on which the other depends.

Recovery accounts for interrupted finalization

The store checks its directory during initialization. It removes temporary snapshot data and inspects the available database files. Where multiple snapshot generations indicate that finalization was interrupted, it can move the previous database into place and replay a remaining WAL. That recovery logic belongs to the snapshot-store format itself.

The directory transitions matter because the operation changes several related files. Persisting bytes, publishing metadata, and incorporating a WAL are distinct steps. Temporary names and startup checks give the implementation a way to recognize intermediate arrangements instead of assuming every prior process exited normally.

Those mechanisms support the redesign's operational purpose, but they do not justify describing all recovery work as independent of database size. Full snapshot transfer still involves a complete file, and large sets of changed pages still create work. The stronger supported claim is that routine incremental snapshot cost no longer has to follow the full database size on every cycle.

The storage simplification affects upgrades

Version 8 always runs SQLite on disk. The changelog explains that late 7.x improvements reduced the write-performance distinction enough that supporting both storage modes added confusion and development cost. Existing commands that pass -on-disk must remove it, because the option is no longer needed or supported.

New cluster joins also use the remote node's Raft address instead of its HTTP API address. The upgrade notes distinguish this from already formed clusters, whose existing membership can permit an upgrade without changing ignored join options. Configuration generators for new nodes need to express the new destination correctly.

The release includes automatic conversion of 7.x snapshots and describes mostly seamless upgrades. It also recommends backing up and presents a new cluster loaded from backup as a conservative alternative. Those are the project's upgrade claims, not evidence that every deployment's configuration will remain valid without changes.

I prefer the design for its specific reduction in repeated work. The first snapshot establishes the base, later snapshots supply compacted changes, and the snapshot store maintains a complete recoverable SQLite file. Resource use then follows the pages affected between snapshots much more directly, while the upgrade makes on-disk SQLite the single operating model.