rqlite restores S3 backups through the elected leader

Downloading a database file is only the first part of restoring a distributed database. If every member independently installs its own copy, the restore process also has to explain how those copies become one agreed state. rqlite 7.16.0 connects the downloaded file to its existing leadership mechanism. The elected leader loads the backup through Raft, while nodes that observe another leader discard their local restore candidates.

The May 5 release changelog lists automatic restoration from S3 alongside readiness controls and a configurable download-failure policy. Together these changes make restoration part of startup without treating a successful object download as evidence that the database is ready. The release follows the automatic S3 backup capability introduced in April.

I like the proportionality of rqlite's architecture, and this is a useful example of it. The restore mechanism uses the store's existing agreement path to install the recovered data. It avoids requiring a separate recovery coordinator, although the startup policy still needs careful interpretation.

Downloading happens before the store opens

The node startup implementation checks whether an automatic restore configuration was supplied after creating the store object. It downloads the configured object into a temporary file, then calls SetRestorePath() with that file's path. The store has not opened at this point, which matters because that method rejects calls after opening.

The download helper first reads and parses the restore configuration. It constructs an S3 client using the configured endpoint, region, credentials, bucket, and object path. A downloader writes the result into a temporary destination. If the operation fails, the helper closes and removes the temporary file through its error cleanup.

This sequencing separates obtaining candidate data from installing it. Successful download leaves a local file available for the store's later decision. It does not immediately replace the live database. The decision depends on the leadership information the store receives after it starts.

The distinction also separates error classes. A configuration file that cannot be read or decoded prevents the helper from reaching a download attempt. A storage request can fail later because the remote operation did not succeed. The optional continue policy applies at a specific point in this control flow, rather than swallowing every possible startup error.

The configuration makes failure policy explicit

The restore configuration includes timeout and continue_on_failure alongside its version and storage type. If the timeout is zero or omitted, decoding supplies a thirty-second default. A duration string can express a different bound for the download context.

This example describes a required restore from an illustrative S3 location. Environment references supply the access key values when the node reads the file.

{
  "version": 1,
  "type": "s3",
  "timeout": "2m",
  "continue_on_failure": false,
  "sub": {
    "region": "us-east-1",
    "access_key_id": "$RQLITE_RESTORE_ACCESS_KEY",
    "secret_access_key": "$RQLITE_RESTORE_SECRET_KEY",
    "bucket": "example-rqlite-backups",
    "path": "database.sqlite.gz"
  }
}

The file reader expands environment variables before parsing JSON. As with the backup configuration, this is startup-time expansion. The S3 configuration definitions also permit an optional custom endpoint in this release. That setting enters the AWS session configuration directly.

When the downloader returns an error, downloadRestoreFile() returns the configured continue flag to its caller. Startup then either logs the failure and proceeds or terminates. File-reading, parsing, and temporary-file creation errors return a false continue value. Setting continue_on_failure therefore does not make an invalid configuration acceptable.

A successful download followed by rejection of the candidate SQLite file is also fatal in startup. That rejection occurs when the store checks the downloaded file, after the download helper has returned. The continue flag should be understood as permission to continue after the downloader's failure path, not as a universal permission to ignore restoration problems.

Compression is detected from the downloaded bytes

The downloader first downloads into its own temporary file. It needs an io.WriterAt destination because that is the interface used by the storage client. After the transfer, it inspects the first three bytes for the gzip signature and resets the file position before processing the content.

If those bytes indicate gzip, the downloader creates a gzip reader and copies decompressed bytes into its caller's destination. Otherwise, it copies the downloaded bytes directly. The object name does not determine which path it takes. A filename ending in .gz is useful to a person, but this code chooses decompression from the content.

The temporary files also explain the local storage requirement. During recovery, the process can hold the downloaded object and the final SQLite candidate at the same time. With a compressed backup, the candidate contains the expanded database. Object storage does not eliminate the local disk space needed to prepare that file.

Download success counters and downloaded-byte statistics belong to this downloader stage. They can establish that data reached the preparation path. They cannot establish that the store subsequently installed that data, since the leadership decision and load happen elsewhere.

The first leadership decision consumes the candidate

The store implementation checks the candidate with db.IsValidSQLiteFile() in SetRestorePath(). It then registers a readiness channel and remembers the path. The store's comments describe the purpose as preventing readiness until a restore has been attempted.

When a leadership notification arrives, selfLeaderChange() checks whether a restore path remains. If another node became leader, it logs that it will not perform automatic restoration and increments the skipped counter. If this node became leader, it calls installRestore() and records success or failure.

In either branch, deferred cleanup removes the candidate file, clears the path, and closes the restore readiness channel. That happens even when the load fails. The mechanism is explicitly a one-shot attempt within that running store. A later leadership change does not retry the same candidate because its path has already been cleared.

That behavior gives the restore configuration a specific role when several newly starting nodes have downloaded the same backup. The winner of the initial leadership decision attempts installation. Followers do not later install their independent candidates merely because they become leader at some future point. Their local copies have already been discarded.

The implementation does not compare the downloaded objects across nodes before making this decision. If configurations identify different backup contents, the selected leader's candidate determines the attempted load. Consistent restore input is therefore a deployment property, while the database code decides which node performs the operation.

Loading uses the existing replicated request

installRestore() opens the candidate and reads its contents into memory. It constructs a LoadRequest carrying those bytes and calls the store's internal load method. The normal public Load() method checks readiness, but the internal method deliberately omits that check. Restoration needs this distinction because its own readiness channel is still open while the load runs.

The internal load path marshals the request as a load command and submits it through Raft. It uses the same kind of replicated command that the store already supports for loading a SQLite file. The candidate enters the database through that path after leadership is established.

The load call waits on the Raft apply result and propagates errors, including loss of leadership. On success it records the applied index. This gives the restore caller an outcome from the replicated load path, beyond the earlier result of downloading an object.

The full read into memory is a separate resource requirement from the earlier file downloads. File-based preparation does not make the entire restore constant in memory consumption. In this version, the installation step explicitly reads all candidate bytes before constructing its request. That matters when assessing how large a recovered database can be for a particular process budget.

Readiness has a similarly precise meaning. Ready() requires a known leader and no outstanding registered readiness channels. The restore channel closes after an attempt or a skip. As a result, readiness alone does not prove a successful automatic restore. A failed attempt can release that condition after the failure has been logged and counted.

I prefer evaluating this feature through those concrete state changes. Automatic download removes a manual staging step, and the leadership hook determines who attempts the replicated load. Recovery success still depends on the load result, which has its own statistics and log messages. The process becoming ready and the recovered data being installed are related events with different evidence in this implementation.