James Ainslie

Preserving a datastore protocol leaves room to replace the engine behind it. Dragonfly's public repository presents that approach on May 31. Its README at the launch-period revision describes a multithreaded, shared-nothing in-memory store implementing Redis and Memcached commands. The design partitions the keyspace across threads so one process can use multiple cores.

The repository also makes its limits visible. Its roadmap reports about 130 Redis commands and targets Redis 2.8 API parity as an initial milestone. Replication and failover remain future work in that roadmap. A familiar client protocol therefore should not be read as a claim that every operational property of an established Redis deployment is already available.

I find the single-machine scaling idea attractive when a cache's workload can use more cores. It offers a different operational choice from immediately distributing the cache across multiple processes or machines. That preference does not eliminate the need to understand command coverage and the license attached to the implementation.

Threads own separate slices of data

The README describes each thread managing a shard of dictionary data. Shared-nothing here means that the keyspace is divided so a thread owns its slice, rather than all threads freely mutating one common dictionary. That ownership is the basis for making additional cores useful.

Single-key ownership is only part of the problem. An operation that touches several keys may need to coordinate across those owners while preserving atomicity. The project says it bases its transactional framework on research into a redesigned lock manager for main-memory databases. Its stated objective is atomic multikey operations without mutexes or spinlocks.

Those statements describe the design claim in the repository. They should not be reduced to a claim that multiple threads alone provide compatibility. A server can parse a familiar command and still implement its ordering incorrectly. Dragonfly's architecture has to account for command semantics as well as distributing the underlying key-value records.

This makes protocol reuse especially interesting. Existing clients can speak to a different internal implementation, but the server must preserve the results and guarantees those commands imply. The roadmap is useful because it separates work already represented by implemented commands from work that the project still intends to do.

Dictionary structure affects memory overhead

The launch-period Dashtable document explains the central storage structure. A selectable database has a primary table for entries and another table for optional expiration information. The design uses a directory of pointers to fixed-size segments, with each segment acting as a small hash table.

When insertion cannot fit within the relevant segment's allowed buckets, the segment splits. A new segment joins the directory, and records from the original segment are redistributed. The document says only two segments participate in that split. Growth therefore does not require allocating another full bucket array for the entire dictionary at each such event.

Inside a segment, regular buckets hold records and stash buckets accommodate overflow. The implementation described in May has a fixed number of slots in each bucket. It tries the home bucket, a neighboring bucket, and the stash locations before concluding that the insertion requires a split.

That detail explains a tradeoff hidden by a simple utilization percentage. A segment can need splitting even while some slots elsewhere remain free, because a particular record has a restricted set of eligible positions. The design favors bounded search and local growth over allowing every empty slot to accept every incoming record.

The document connects this structure to reduced metadata and more localized allocation. It also explicitly warns that improving dictionary overhead cannot drastically reduce every workload's total memory consumption. When values dominate the allocation, saving dictionary metadata has a different proportional effect than it does for many very small values.

Expiration can avoid unnecessary growth

A full segment is also a useful place to look for expired records. The document describes scanning that segment before splitting it. Deleting expired entries can create enough space for the insertion, avoiding growth that would otherwise retain data no longer useful to the application.

The same design includes background scanning alongside expiration work done during other operations. Expiration therefore participates in memory management, rather than acting solely as a check on whether a requested value should be returned. This distinction matters for a store where expired records can consume space even if no client asks for them.

The README documents limits too. Expiration ranges are approximately four years, and long millisecond deadlines are rounded according to a stated threshold. The differences document at the same revision records those semantics along with string-size and index limits.

These are examples of compatibility details a protocol label does not communicate. An application might use only a few common commands while depending on the precise permitted range of one argument. The repository's explicit differences are therefore as relevant to adoption as the larger command checklist.

Snapshots use a different implementation strategy

The Dashtable document says Dragonfly's save procedure is asynchronous and maintains point-in-time snapshot guarantees without forking. It uses table metadata to support that implementation. This addresses a different source of memory pressure from ordinary dictionary overhead.

Its comparison experiment populates data, applies update traffic, and initiates snapshots while recording memory use. The author explains measuring through cgroups because a forked process can share some memory with its parent. Counting each process independently would not establish the same total memory footprint.

The measurements are project-published experiments with described inputs and machines. They are evidence about those runs, not an independent guarantee for arbitrary value sizes or write patterns. The useful architectural point is that Dragonfly chooses a snapshot mechanism that does not require a child process created through fork.

I consider that choice relevant to the operational appeal of scaling one process up. A large-memory instance needs an account of snapshot behavior as well as request throughput. The design document connects the snapshot procedure to the storage metadata instead of treating persistence as an unrelated background task.

BSL places limits on production use

The license in the May revision is Business Source License 1.1. The general grant permits copying, modification, redistribution, and non-production use. Production permission comes through an additional use grant with specific conditions.

That grant is narrower than unrestricted use of publicly readable source. It permits use as part of the user's own product or service provided that product or service is not an in-memory datastore product or service. It also restricts making the licensed work available as a competing service under the license's definition.

Calling the project open source without qualification would obscure those limits. Source availability lets engineers inspect and modify the implementation, but it does not grant every production deployment or commercial offering the same permission. The additional grant is part of the launch proposition, alongside the architecture.

The license also specifies a change date and an Apache 2.0 change license. That is a term governing the version's licensing schedule. It does not make the current grant equivalent to Apache 2.0 on the day the repository becomes public, and it should not be used to predict how later releases will be licensed.

Compatibility and deployment have explicit limits

The README's Memcached support excludes cas at this point. Its Redis checklist also leaves several command families incomplete. Replication remains a milestone, so the presence of familiar connection commands or an accepted command name must not be confused with a completed high-availability design.

The launch-period server also requires Linux 5.10 or later because it uses io_uring for I/O. That is a concrete deployment requirement separate from API compatibility. A client that can issue a Redis command cannot make an unsupported host kernel satisfy the server's I/O needs.

The two wire protocols also have separate configuration choices. The README says the Memcached port is disabled by default and enabled through its own argument. A deployment using Memcached therefore needs to select that listener explicitly, even though the same engine implements both sets of operations.

Dragonfly's initial proposal is consequently specific. It combines per-thread data ownership with a segmented dictionary and a forkless snapshot implementation, while keeping familiar client protocols. Its public roadmap and license state the boundaries of that proposal. Those materials let an engineer inspect both the intended scaling mechanism and the functionality still missing from the May implementation.