October 24, 2017·4 min read·engineeringsqlitestoragetransactions

SQLite delegates eligible atomic commits to F2FS

SQLite 3.21.0 can use F2FS atomic writes to avoid creating a rollback journal for eligible commits. The optimization requires both a build option and a filesystem capability reported through the VFS.

A database engine normally has to arrange its writes so that a failed transaction cannot leave only part of the change in the database. If the filesystem can commit a batch of writes atomically, some of that work can move below the database. SQLite 3.21.0, released on 24 October, adds an optional path that uses atomic-write capabilities in F2FS to reduce transaction overhead.

The change is conditional in two separate ways. SQLite must be built with SQLITE_ENABLE_BATCH_ATOMIC_WRITE, and the storage path must report the required capability. Installing the release does not make every transaction on every filesystem use the new method. The interesting part is how the pager recognizes when it can delegate part of its commit protocol.

I like this approach because the application can retain its transaction interface while SQLite adapts to the storage beneath it. The benefit comes from a specific guarantee that the lower layer supplies. It does not require the application to approximate atomicity by issuing its own sequence of file writes.

The VFS reports a storage capability

The 3.21.0 pager source checks for SQLITE_IOCAP_BATCH_ATOMIC in the device characteristics returned by the virtual filesystem. This flag is separate from the older atomic-write optimization for an individual database page. The source describes distinct eligibility rules for those two cases.

For the single-page optimization, the reported atomic-write characteristics and sector size must fit the database page. For batch atomic writes, the capability bit identifies support for the batch operation. Keeping those cases distinct matters because a transaction can modify more than one page. A guarantee about writing one page does not establish a guarantee about a collection of pages.

The Unix VFS implementation contains the F2FS-specific control operations. Under Linux, with the build option enabled, its file-control handler translates SQLite's begin, commit and rollback requests into filesystem ioctl calls. Failures return corresponding SQLite I/O error codes.

This division keeps the pager's commit decision separate from the platform-specific operation. The pager asks for an atomic batch through SQLite's file-control interface. The Unix implementation knows which filesystem operations implement that request. An application using SQL transactions does not issue the F2FS ioctls itself.

Eligibility belongs to a particular commit

The pager computes a bBatch condition for the commit path. It requires the batch capability, an in-memory journal and synchronization to remain enabled. It also requires the absence of a master-journal name. These tests make the decision narrower than a global assertion that the database is on supported storage.

The in-memory-journal condition is consequential. The source also contains a path that creates the rollback journal when dirty pages must spill. A transaction whose execution has already required that path is different from one whose journal remains in memory until commit. Storage support alone therefore does not determine whether a particular transaction takes the optimization.

The master-journal condition separates this path from a commit requiring that additional coordination. The code explicitly creates a journal when a master journal is present. Describing the feature as removal of all rollback journals would miss both this condition and the cases that materialize a journal before commit.

I find these conditions useful because they make the optimization explainable. They connect the outcome to transaction state and storage capabilities. An application can benefit without changing its SQL, but that does not mean every execution of a transaction has an identical physical commit path.

Beginning and completing the batch

When the batch path is selected, the source comments say that no rollback journal is created. Before starting the filesystem operation, the pager closes its in-memory journal. Its comment explains that attempting to play that journal back into the database after the following operations would be unsafe.

The pager then sends SQLITE_FCNTL_BEGIN_ATOMIC_WRITE. If that succeeds, it writes the dirty page list. Successful page writes lead to SQLITE_FCNTL_COMMIT_ATOMIC_WRITE. A failure while writing pages leads to a rollback request. Errors from these stages propagate through the commit path.

This is a concrete reduction in work at the database layer. SQLite can avoid creating the usual rollback journal for the eligible commit because the batch operation supplies the atomic-write mechanism. The implementation still checks errors and still has a commit protocol. The work changes location and form because the filesystem exposes a suitable operation.

The public release notes describe greatly reduced transaction overhead when the capability is available. They do not supply a universal speedup for arbitrary applications. The source explains why the result depends on eligibility, including whether a transaction retained its journal in memory. A numerical performance claim would require a workload and storage configuration that the announcement does not provide.

Other storage behavior remains explicit

Version 3.21.0 also changes the synchronization used after writing a WAL-reset header. That operation now uses checkpoint synchronization settings, including checkpoint_fullfsync on macOS. This is a separate storage change from the F2FS batch path. Combining them into one general claim about faster commits would obscure the different mechanisms and platforms involved.

The release allows ATTACH and DETACH inside a transaction. That language change should also remain separate from the absence-of-master-journal condition in the batch implementation. Accepting a statement inside a transaction does not establish that every resulting commit qualifies for a particular storage optimization.

Other changes improve observability and correctness. Integrity checks detect additional obscure row corruption and return error text for corruption in records. The source identifier tries to signal accidental source edits by changing its final characters, while the notes explicitly say a forger can defeat that detection. These are bounded claims about particular checks, not substitutes for the commit mechanism.

For an embedded database, the F2FS work is valuable because it preserves the SQL-level transaction while exploiting a lower-level operation that fits it. The released code gives the exact boundary. The build must include support, the VFS must report the capability, and the individual commit must satisfy the pager's eligibility tests before SQLite requests an atomic batch.