rqlite adds scheduled SQLite backups to S3

Keeping a replicated database available and retaining a separate copy of its contents require different mechanisms. rqlite already has a distributed store, but its 7.15.0 release adds a scheduled path for copying database contents to Amazon S3. The April 28 changelog identifies automatic S3 backups as this release's new capability.

The implementation is deliberately small. A timer asks the store for a SQLite backup file, optionally compresses it, and passes the resulting bytes to an S3 client. A checksum comparison avoids uploading the same output again. These choices make the mechanism understandable, but they also define its operational scope. The service copies complete database files to a configured object path. It does not implement a history of transaction-level recovery points.

I value rqlite's ability to supply distributed relational storage without demanding a large collection of supporting services. An integrated backup scheduler fits that preference. Understanding which work it removes, and which recovery decisions remain outside it, is more useful than treating the presence of an S3 destination as a complete backup policy.

The schedule is part of the node process

The startup code starts automatic backups after creating the cluster and reporting that the HTTP API is available. The -auto-backup flag names a JSON configuration file. If the flag is absent, startup returns without creating an uploader. If the file cannot be read or parsed, the process reports the failure through its fatal startup path.

After parsing, startup creates an S3 client and an uploader, then runs the uploader in a goroutine. It also registers an auto_backups status provider with the HTTP service. The backup loop therefore has a lifetime tied to the node process, without an additional scheduler executable to deploy.

The uploader creates a ticker using the configured interval. It waits for a tick before attempting an upload. Startup does not trigger a separate immediate backup in this implementation. When the node receives its shutdown signal, the startup code cancels the backup context as part of closing its services.

The interval describes when the process attempts work. It does not state that an upload will complete at precisely that cadence. Backup generation, compression, and the S3 operation all occur within the loop's upload call. Their duration affects when the loop can process another timer event.

Configuration describes one destination

The configuration definitions give the file a version number, storage type, interval, and nested storage settings. The supported storage type is s3. Its interval parser accepts a duration string, which makes a value such as 1h less ambiguous than a bare numeric value.

An illustrative configuration for this version has the following shape. The environment variable references are placeholders for deployment-supplied credentials, not values to copy literally into a live secret store.

{
  "version": 1,
  "type": "s3",
  "interval": "1h",
  "sub": {
    "access_key_id": "$RQLITE_BACKUP_ACCESS_KEY",
    "secret_access_key": "$RQLITE_BACKUP_SECRET_KEY",
    "region": "us-east-1",
    "bucket": "example-rqlite-backups",
    "path": "node-a/database.sqlite.gz"
  }
}

The file reader expands environment variables before JSON decoding. That permits credential values to enter through the node's environment while the configuration retains references to them. The expansion occurs when startup reads the file. This code does not implement a periodic reread of credential values.

Compression is enabled unless no_compress is true. The object path is configured independently of compression, so the .gz suffix in this example communicates the chosen format to a person retrieving the object. The uploader does not derive the object name from its compression setting.

The S3 client constructs an AWS session using the configured region and static access key credentials. This version explicitly calls NewStaticCredentials. A description of this release should therefore avoid assuming that its backup client automatically uses every credential-discovery mechanism available elsewhere in the AWS SDK.

The backup becomes a temporary file

The uploader's data provider accepts a destination path. The store implements that interface by calling its database backup method with the supplied path. This creates the SQLite backup that the uploader subsequently processes. The S3 client receives the resulting file contents, rather than a stream of SQL statements assembled by the scheduler.

Pull request 1232 is useful context for this design. It changes the provider interface to write into a caller-specified file and replaces an in-memory compression buffer with file-based compression. That keeps the compressed output out of one growing application buffer, although the operation still needs temporary filesystem space.

With compression enabled, the uploader creates another temporary file and writes gzip output into it. Once compression finishes, it renames that file over the original upload path. The cleanup paths request removal of the temporary files after the upload operation. This sequence means the node needs room for both the uncompressed backup and compressed output while compression is in progress.

That resource requirement follows directly from the mechanism. Moving the final copy to object storage does not remove the local work needed to construct it. Backup scheduling therefore introduces disk activity and temporary storage requirements on the node that runs it, even though the retained object lives elsewhere.

Unchanged files can avoid a network upload

After generating and optionally compressing the file, the uploader computes its SHA-256 hash. It compares that hash with the last successfully uploaded hash held in the uploader instance. Equal hashes increment a skipped-upload counter and return without sending the file to S3.

The placement of the comparison matters. It saves the S3 transfer for identical output, but it happens after backup generation and compression. It does not use the checksum to avoid creating a local copy in the first place. The service must first obtain the candidate bytes before it can compare them.

Failures during file preparation return before the storage-upload counters change. For example, a provider error or a compression error exits the upload method before it calls the storage client. The outer loop still logs that error. Reading only the failed-transfer counter would therefore omit failures that prevented a transfer from starting, even though those failures also prevented the scheduled backup from reaching S3.

Only a successful storage upload replaces the remembered hash. A failed upload leaves that state unchanged, so another attempt can still send the candidate file. The remembered hash lives in the running uploader object. Restarting the node creates a new uploader without the previous process's checksum history.

The status provider reports the destination, interval, and compression setting, together with the last successful upload time, duration, and checksum. Separate counters track successful, failed, and skipped uploads. A skipped interval therefore should not be confused with a newly completed transfer. The last successful time changes when the upload call succeeds.

Node selection remains a deployment decision

The uploader interface accepts an optional function that determines whether uploading is enabled. If none is supplied, the loop substitutes a function that always returns true. In 7.15.0, node startup calls Start(ctx, nil). There is no leader-only condition in that wiring.

This detail changes how the configuration should be understood in a multi-node deployment. Giving every node the same backup file enables every corresponding uploader. The code does not first elect a single backup producer. If those files name the same bucket and path, the upload requests target the same object key.

The S3 client supplies the configured key directly to each upload request. It does not add a timestamp or node identifier. A deployment that needs distinct destinations must express those destinations in configuration. Likewise, a requirement to retain multiple historical copies needs a storage policy or additional mechanism beyond this uploader's fixed path.

I prefer making those boundaries explicit because they keep a useful convenience from acquiring guarantees its code does not provide. The integrated service automates complete database copies and provides observable upload results. Decisions about which node produces them and how long earlier copies remain available still belong to the deployment.

The upstream change also includes an end-to-end compression check. In pull request 1232, the test inserts a thousand rows, retrieves and decompresses the uploaded object, opens it with SQLite, and checks the matching row count. That tests the content produced by the compression path, beyond merely checking whether the S3 call returned successfully. It is the relevant relationship for this feature, since the remote object must contain a usable database copy for a later recovery operation.