Litestream changes the representation of SQLite backups
Litestream 0.5 adopts LTX and compaction to reduce repeated page work during restores. The format change also separates new backups from the older WAL-segment restore path.
A busy database can update the same page repeatedly while a backup records every version. Restoring that history then does work the final database no longer needs. Ben Johnson's 2 October announcement of Litestream 0.5 explains how a new LTX-based storage format lets the backup system compact those repeated page changes before recovery needs them.
The announcement follows the 0.5.0 release published on 30 September. The October article supplies the design explanation and upgrade contract. This change is substantial because a backup's representation determines which files a restore needs and which version of the recovery tool can interpret them.
I like SQLite because a full application can use a relational database without operating a separate database server. This site's workload gives me no reason to replace it. A tool that handles off-machine recovery can complement that choice, but its restore contract deserves the same attention as its ability to upload changes.
Repeated page changes accumulate work
Litestream runs beside an unmodified SQLite application. Its release-tag README describes a standalone disaster-recovery program that incrementally replicates changes. The October announcement explains the page-level problem using a table with an increasing primary key. New inserts repeatedly touch the rightmost leaf page of its B-tree.
The repeated writes are ordinary database activity. They become a backup concern when the recovery representation preserves many versions that a later restore must process. The final state needs the newest applicable version of each page. Earlier versions remain relevant to older recovery points, but they need not all be replayed to reconstruct every later state.
That distinction separates retaining recovery history from repeating all historical work during a restore. A representation can preserve useful points in time while arranging the data so that a particular restore selects a smaller set of inputs. Litestream's format change is aimed at that arrangement.
The announcement connects this work to LiteFS, Johnson's replication system. LiteFS needed a transaction-aware interchange format, which led to LTX. Litestream adopts the format's ordered page ranges and compaction support without turning its backup process into the same deployment model as a FUSE-based live-replication system.
Compaction removes superseded page versions
An LTX file represents an ordered range of pages and supports compaction with other LTX files. When several files contain the same page, the newest applicable version wins. Compaction can produce a file without duplicate versions of that page for the represented interval.
The important qualifier is the interval. A restore to an earlier point cannot simply use every newest page ever backed up. It must select files that represent the requested history. The compaction hierarchy makes larger spans convenient while retaining smaller spans needed to assemble a particular recovery point.
The launch describes levels covering thirty seconds, five minutes and an hour. Each larger level compacts files from the smaller level. These intervals are implementation defaults described by the announcement, not a promise that every application generates the same number of backup files or has identical restore latency.
Johnson estimates that a restore can use roughly a dozen files on average. That is the project's reported expectation. The mechanism behind it is the reduction in repeated page versions and object fetches, not a universal timing result. The amount of data a restore actually transfers still depends on the database and selected recovery point.
Litestream performs compaction itself. It does not require SQLite to replay WAL files to carry out that work. This makes the archived representation an active part of the backup system, with its own processing separate from the live database's normal checkpointing.
Transaction identifiers replace generations
Earlier Litestream backups used generations to separate histories when the backup process lost WAL continuity. The October explanation describes a process that notices a gap and resynchronizes. Multiple independently evolving generations make locating the desired database state more complicated.
The LTX design instead takes another snapshot when continuity breaks and advances the transaction identifier. The resulting history uses monotonically increasing transaction IDs. Commands refer to those identifiers in place of the older generation, index and offset arrangement.
This changes the coordinates used to describe a restore. Operational instructions that record only an old generation-and-offset location do not automatically name an LTX transaction. A format migration therefore affects runbooks and tooling that inspect the backup history, even where the main replicate and restore command workflow remains familiar.
The command used to inspect stored changes also changes name. litestream wal becomes litestream ltx. That is a visible expression of the new representation. The older name would imply that the archived objects remain the same WAL segments, which they do not.
Old recovery data retains its old reader
Version 0.5 cannot restore from the WAL-segment files produced by 0.3.x. The announcement says starting the new version leaves those old files intact and stores new LTX files in an ltx directory on the replica. Keeping old data is useful, but retaining it does not give the new executable the ability to read it.
An upgrade consequently creates a distinction between recovery histories. A restore from an old backup needs the older compatible tool. New LTX history needs the new path. This follows directly from the stated incompatibility and is more precise than describing the upgrade as a transparent continuation of every existing restore procedure.
The article calls the configuration backward compatible, but immediately identifies a new restriction. A database gets one replica destination. A previous configuration relying on several destinations cannot retain that arrangement unchanged. The source explains the restriction as a simplification of divergent replica histories and conflict handling.
I prefer that restriction to an ambiguous promise about which destination contains the authoritative recovery history. It is still a real capability change for users with multiple destinations. The operational benefit and the removed configuration option should both appear in an account of the release.
The format prepares for selective page access
The LTX library now compresses pages separately and records an index at the end of the file. That permits individual-page access without decompressing an entire large file. The October article describes a future read-replica VFS using this ability, but presents it as work in progress rather than the shipped meaning of the backup upgrade.
The release also adds NATS JetStream as a replica type and switches to a pure-Go SQLite dependency for builds. Those changes broaden integration choices. They do not remove the format boundary between old WAL segments and new LTX history. For an upgraded deployment, the recovery tool and the archived representation must remain a compatible pair, including when the required recovery point precedes the upgrade.