Dragonfly adds experimental SSD storage for string values

Memory consumption in a cache includes the values that applications store and the space that the allocator cannot currently reuse. Dragonfly addresses both in its 0.13 release, published on January 1. Experimental SSD tiering moves string payloads out of memory. Improved active defragmentation addresses wasted space within memory. These changes affect different parts of the capacity problem, so their benefits depend on different properties of the dataset.

I am interested in Dragonfly because scaling a single process across a larger machine can avoid some of the operational work of distributing a cache. SSD tiering extends that interest to storage capacity.

The useful question is how much application data can leave memory while the server continues to locate keys and execute commands. The implementation gives a more specific answer than the phrase SSD support suggests.

The dictionary stays in memory

The tagged README describes a shared-nothing architecture. Worker threads manage separate slices of the keyspace. These internal shards let a process use multiple cores, with a transaction framework coordinating operations that involve multiple keys. The architecture already divides responsibility for data inside one server before SSD storage enters the design.

The tiered storage implementation follows that division. It constructs a backing filename using the configured base path and the shard identifier. Each storage object works with its owning database slice. The server retains the dictionary entry and changes the representation of its value after a successful offload. That representation records an external offset and the original payload length.

This explains why the feature cannot make an arbitrarily large keyspace fit in a fixed amount of RAM. Keys and dictionary structures still consume memory. Moving a substantial string payload can save much more than moving a small one because the retained lookup information remains necessary in both cases. A workload dominated by large cached documents therefore presents a different opportunity from a workload dominated by tiny counters. That is a consequence of the retained dictionary, not a benchmark claim.

An external value still belongs to the database. The implementation exposes a read method that accepts its file offset and length and asks the I/O manager to retrieve those bytes. A successful offload changes where the payload lives. It does not require the application to reconstruct the value from some other service when the key is read.

Value sizes determine the write path

The header at the release tag sets a minimum eligible blob length of 64 bytes. The implementation also requires a string that is neither already external nor waiting for an outstanding I/O operation. These conditions prevent the server from submitting another offload for a value whose storage transition is already in progress.

Within the eligible population, the code uses separate paths for smaller and larger values. Smaller strings enter size bins. The implementation packs several values into a 4,096-byte block, reserving space for per-entry hashes as well as payloads. A bin accumulates enough pending entries to fill its batch before it submits the write. This layout shares a disk block across several values instead of assigning each small string an entire block.

Batching has a practical consequence for capacity estimates. An eligible value does not necessarily leave memory immediately. Its bin may still be waiting for more entries. The implementation can also skip an offload attempt when writes are saturated. Counting every eligible byte as an immediate reduction in resident memory would overstate what this code guarantees. Eligibility describes a permitted transition, while batching and available I/O determine when the transition happens.

The larger-value path begins above the small-bin cutoff of 2,032 bytes. It allocates an external range and rounds the write buffer up to the block alignment. The original blob length remains distinct from that aligned size. This matters because the bytes occupied in storage need not equal the useful bytes supplied by the application. Alignment and packing decisions influence the space saved in RAM and the space consumed on the device separately.

The storage allocator can grow the backing file asynchronously. In the small-value scheduling path, allocation above 85 percent of capacity triggers a request to grow by 256 MiB. These constants are properties of this tagged implementation. They explain why a configured prefix identifies a managed storage file, rather than a fixed-size file that the operator fills manually with serialized records.

Writes complete after scheduling

An SSD write does not become an external dictionary value merely because the server submitted it. In the batched path, entries carry an I/O-pending marker while the request is outstanding. The completion handler checks which entries still belong to that request, updates the surviving entries, and clears their pending state. It also adjusts the recorded object-memory and tiered-storage statistics when it externalizes an entry.

Cancellation is part of the same mechanism. The code can remove an entry from a pending bin or erase it from the map of enqueued entries. That distinction reflects the difference between waiting for a batch and belonging to a write that the server has already submitted. Updates and deletions can overlap the lifetime of an asynchronous request, so the request needs bookkeeping beyond a file offset.

The completion logic can abandon a batch if too few of its original entries remain usable. It frees the allocation and undoes the pending state for entries that still belong to the request. A failed write also takes an undo path. These branches show why offloading is a storage transition with intermediate states. A submitted request does not establish that its original payloads have all become external values.

Pages containing several external values need another form of bookkeeping. The implementation keeps a reference count for such pages and returns a page to the allocator when its final reference disappears. Values written in their own aligned allocation follow a different free path. The packing that reduces small-value storage overhead therefore also requires the server to track the lifetime of the shared page.

Enabling the experiment

The release notes provide the startup option directly. With an existing directory on the intended device, the relevant invocation is:

./dragonfly --spill_file_prefix=/mnt/nvme/dragonfly-tier

The argument is a base path. The tagged implementation appends a shard-specific suffix and the .ssd extension. That directory must consequently be appropriate for the files the running process creates. This example enables the experimental tier and does not specify a recovery policy or promise that every value moves to disk.

The implementation exposes statistics for tiered reads and writes, aborted writes, skipped flushes, and storage allocation. These counters describe different stages of the mechanism. A submitted write and a completed offload are distinct events, and a skipped attempt leaves its payload in memory. Their distinction is useful when explaining why the memory footprint does not fall in direct proportion to the eligible strings.

I regard the string-only scope as a useful constraint on this first implementation. An opaque byte sequence gives the server a bounded payload to externalize. The release does not establish equivalent SSD behaviour for hashes, lists, or sorted sets. Applications using those data types should account for their memory independently instead of applying a string-tiering estimate to the whole dataset.

Memory reclamation remains a separate problem

The active-defragmentation change addresses external fragmentation, according to the release notes. Moving payloads out of the allocator and making allocator pages reusable are related capacity concerns, but they are different operations. A database can have fewer live payload bytes while still retaining pages whose remaining allocations prevent reclamation. SSD capacity does not by itself explain the resident memory of the process.

The release-tagged README gives another relevant distinction. Dragonfly describes snapshotting without a process fork, and its dictionary design supports incremental growth and traversal during changes. Those mechanisms address snapshot and dictionary behaviour. Tiering adds a file-backed representation for selected values within that existing design. None of these mechanisms eliminates the metadata required to keep a key addressable.

The asynchronous implementation also needs temporary write buffers. A value awaiting completion can still occupy memory even though an SSD request already contains a copy of its bytes. The larger-value path allocates an aligned buffer before submission. That temporary allocation belongs in an explanation of peak memory during offload, alongside the eventual reduction in resident payloads. Steady-state payload size and the memory needed to move data between representations are separate quantities.

For my purposes, the useful result is a more detailed model of vertical scaling. CPU capacity comes from internal sharding, while string payload capacity can extend into SSD files. Available RAM must still cover dictionary structures, resident values, and work in progress. In this release, the batch sizes and asynchronous completion rules explain how that division changes as the server runs.