Dragonfly snapshots can include values offloaded to SSD

A snapshot traversal cannot serialize a string whose bytes are on SSD as though the value were still resident in memory. It must arrange to read those bytes, while respecting the traversal's own restrictions on blocking. Dragonfly's tiered snapshot implementation handles this by collecting delayed entries. The traversal schedules reads, and a later stage waits for the results before serializing them.

That implementation arrives in Dragonfly 1.19.0, released on June 5. The release also fixes memory growth during replication and an RDB compatibility problem affecting Redis 6.x. These changes all concern moving a consistent representation of database contents through a pipeline, though the failures and remedies differ.

I am interested in tiering because it extends Dragonfly's argument for using one larger machine. The operational value increases when data placement works with snapshotting. A value's storage location should not prevent it from entering the recovery artifact that represents the database.

Snapshot traversal schedules work it cannot await

Before this change, Dragonfly explicitly prevented saving in tiering mode. The patch removes the save-path rejection and a snapshot-scheduling check that exited when tiering was enabled. Removing those guards is only the visible part of the change. The serializer also needs the external value's contents, not its current disk-location metadata.

The new branch in entry serialization checks whether the value is external. For an ordinary resident value, the existing serializer still handles the entry directly. For an external value, Dragonfly requests a tiered read and stores a future for the resulting string. The delayed entry also retains the key, database identifier, and expiry time needed to serialize it later.

That separation follows a constraint documented in the code. The bucket serialization path cannot block, but the function that pushes serialized data to the channel can. At that later point, Dragonfly waits for each delayed value and writes the full entry through the serializer. This lets the snapshot include the string contents while preserving the nonblocking requirement of the earlier traversal.

The distinction between scheduling and awaiting is important. Simply adding a synchronous disk read to the original traversal would ignore why that code cannot block. The implemented approach instead moves the wait into a stage that permits it. The future connects those two stages without treating the external pointer as a persistent representation of the value.

Storage movement pauses during the snapshot

The patch also checks whether a snapshot is in progress before running background offloading. While that condition holds, the offload loop returns. Another guard prevents a fetched value from being installed back into memory through the relevant fetch-reporting path during snapshotting. These constraints reduce changes to value placement while the serializer gathers its data.

This is a concrete tradeoff in the initial implementation. Snapshotting gains access to tiered values, while the background mechanism for moving values out of memory temporarily yields to that work. The source supports an explanation of this coordination, but it does not establish that snapshotting has no effect on tiering activity or I/O demand.

The regression test creates string values of different sizes and waits for offloading activity to settle. It then saves a Dragonfly-format snapshot, flushes the database, loads the snapshot, and compares the recovered contents with the original capture. That sequence tests restoration of the values, rather than merely checking that a save command returned success.

I value that recovery-oriented test because the purpose of a snapshot extends beyond producing a file. Here, the assertion covers the representation after it has passed through storage and reloading. It also avoids treating the SSD backing location as the complete recovery mechanism by itself.

Full sync must allow its channel to make progress

The first replication correction addresses memory growth during full sync for commands that do not use automatic journaling. Its description explains that those commands cannot preempt while writing the journal. A callback-completion path later uses a no-op opcode to trigger writing to the synchronization channel.

Full-sync mode ignored that no-op. Consequently, the trigger intended to move buffered work onward did not perform its role in this mode. The patch allows writing to the channel when full sync receives the no-op. The relevant distinction is the opcode's coordination purpose, even though its name suggests an operation with no data effect.

A separate full-sync ordering fix addresses a deadlock. The full-sync fiber saved the header before reading the body from a bounded channel. Header generation needed global data obtained through a shard callback. Meanwhile, a transaction at the head of the shard queue could be blocked trying to write to the already full snapshot channel.

Those dependencies formed a cycle. Header generation waited for the shard, the shard waited for channel space, and consuming the channel waited for header generation to finish. The correction obtains the global data and saves the header before starting snapshotting of the shard data. The pipeline therefore resolves the header dependency before producers can fill the channel.

This illustrates why a bounded queue alone is not a complete solution to buffering problems. Backpressure limits the queue, but its placement can block the work needed to start the consumer. The fix changes stage ordering so that the bounded channel can fulfil its purpose without stopping the prerequisite callback.

Disk snapshots must exclude replication-only records

The RDB compatibility correction removes a Dragonfly full-sync-cut opcode from disk snapshots. Its description identifies that record as incompatible with the RDB format. The added test loads a Dragonfly-generated RDB snapshot using a Redis server, which checks the interchange direction affected by the bug.

Snapshot serialization and replication can share machinery while still requiring different output records. A control marker useful during full synchronization is not automatically valid in a disk file another engine will load. Version 1.19 keeps that marker out of disk snapshots, alongside adding the delayed-read path that turns tiered strings back into serializable values.