rqlite removes in-memory staging for on-disk startup

An optimization that constructs an on-disk database in memory first can make startup depend on having enough memory for that intermediate copy. rqlite previously used this approach to accelerate reconstruction, with a switch for databases that could not tolerate the memory requirement. The 7.21.4 release removes that choice because recent changes have reduced the performance gap between its on-disk and in-memory modes.

The July 8 changelog attributes the improvement to SQLite WAL mode and synchronous-off operation. The immediate change is a simpler startup path for an on-disk database. Its significance becomes clearer when connected to how rqlite reconstructs SQLite state from the replicated history it retains.

I value rqlite's proportionate use of distributed systems machinery. Removing a startup setting after eliminating its purpose fits that preference. The useful question is which intermediate state disappeared, because the database still has recovery work to perform when it opens.

Why the old control existed

Pull request 1336 removes the StartupOnDisk field from the store. Its deleted comment explains the earlier arrangement. Restarting an on-disk database could be slow, so rqlite could first reconstruct it in memory and then move it to disk. The control let users bypass that optimization when the temporary memory requirement prevented startup.

The old path also had conditional behavior based on whether snapshots and commands were present. During replay, the store could wait until it reached the last command known at opening, serialize the in-memory database, close it, and create the on-disk database from those bytes. That introduced an additional transition inside the recovery process.

In the new store implementation, opening follows the configured database mode directly. An in-memory configuration creates an in-memory database. An on-disk configuration creates the on-disk database before Raft initialization. The removed replay branch no longer performs the late conversion from memory to disk.

This does not remove the in-memory operating mode. It removes the use of that mode as temporary staging for an on-disk configuration. That distinction matters when reading the short changelog entry, which could otherwise sound as though all database storage choices had disappeared.

June changed the cost of reconstruction

The versioned changelog records two earlier changes that explain the July simplification. Version 7.20.4 changed SQLite's synchronous mode to off for on-disk operation. The maintainer reported at least a threefold write-performance improvement in that testing. Version 7.21.1 then enabled WAL for on-disk databases and reported an approximately thirty-percent improvement in its tests.

Those figures describe the project's reported testing. They are not a promise that every workload receives the same increase, and they should not be combined into a general performance guarantee. Their relevance here is the maintainer's stated reason for retiring a separate startup optimization.

The durability explanation is specific to rqlite. The changelog says that rqlite deletes its SQLite database on startup and rebuilds it from the Raft log. SQLite's local database file therefore does not have the same recovery role as the retained Raft state. Reducing SQLite synchronization does not mean the system has no persistent history.

The current createOnDisk() helper makes the reconstruction behavior explicit. It removes existing SQLite files before opening the database. If supplied with recovered bytes, it writes those bytes to the configured path before opening it. An operator should therefore distinguish the local SQLite representation from the Raft data that permits reconstruction.

Existing commands continue to parse

The patch changes the help text for -on-disk-startup to say the option is ignored. It leaves the flag registered, but removes the assignment that previously copied its value into the store. An existing startup command containing the option can therefore remain syntactically acceptable while the option ceases to control behavior.

The ordinary -on-disk setting still selects on-disk SQLite operation. A minimal illustrative command remains straightforward.

rqlited -on-disk /var/lib/rqlite

The data directory in this example belongs to the whole node. Selecting on-disk SQLite does not make its database file an independent substitute for all the node's other retained state. The startup code still creates the Raft system, and recovery still uses the history and snapshots available to that system.

The patch also removes startup_on_disk from the store's status output. Tools inspecting that field should account for its removal. Its disappearance corresponds to the removal of an internal control, rather than a runtime report that the on-disk setting itself is false.

Fewer transitions during recovery

The restore path now selects the configured storage mode directly as well. For an on-disk configuration it calls createOnDisk() with restored bytes. It no longer needs to preserve the old decision about whether to construct an intermediate in-memory database and postpone the switch.

I prefer this arrangement because it reduces the states needed to understand an on-disk node opening its database. The configured storage mode applies during reconstruction, and the obsolete flag no longer changes that process. Recovery still depends on the retained Raft information, while the temporary in-memory database and its later serialization step are absent from this startup path.