Dragonfly ends the year with more ways to extend one machine

Dragonfly's growth story is becoming more specific than replacing a Redis process with a faster executable. Its multicore engine now sits alongside SSD tiering and Swarm, with the latter supplying a managed path across machines. Those additions address different constraints, so combining their advertised capacities into one headline obscures what each changes.

The December 18 annual review reports that the project has passed 29,500 GitHub stars and describes a year of engine improvements and cloud expansion. It is a retrospective, not a new database release. The star count records interest in the project, while the technical announcements explain what someone considering Dragonfly can actually choose to run.

I prefer Dragonfly when a cache has outgrown one Redis process and a larger machine is still simpler than a cluster. This year's additions are relevant to that preference because they preserve a useful question about capacity. Which limit requires another machine, and which can be addressed inside the one already running the cache?

Multicore execution remains the starting point

The architecture document shipped with 1.35.1 describes a single process with multiple threads and a database divided into internal shards. Each shard is owned and accessed by one thread. Connection fibers coordinate work with the threads owning the relevant data through messages.

That ownership model explains how a single endpoint can use multiple cores without requiring a separate server process for each internal shard. A connection's work and a key's storage do not have to reside on the same thread. The coordinator hides that placement from the client, including the coordination needed for commands touching multiple keys.

Fibers provide asynchronous execution within each thread. They do not themselves create parallel CPU execution, which comes from the threads. The architecture document makes this distinction explicitly and also acknowledges CPU and latency costs in the coordination layer. The argument for a larger node rests on using its resources through this execution model, rather than treating vertical growth as extra RAM attached to one command execution thread.

SSD tiering changes the payload budget

The November tiering announcement describes keeping keys and associated metadata in memory while moving value payloads into per-thread files. An external value's in-memory entry identifies its storage location. The announcement limits tiering support to string values, so this is not a claim that every Redis-compatible data structure can be moved out of RAM.

A cooling state retains an in-memory copy of a value also written to storage. That copy can serve reads, but it can also be discarded when memory is needed without first waiting for a disk write. This adds a useful state between a value held only in RAM and a value that requires a storage read.

The tradeoff depends on access patterns and metadata as well as total payload size. Keeping lookup information in memory gives the server a direct path to the stored bytes, but it does not remove the memory cost of having many keys. Extending payload capacity and extending the maximum number of inexpensive in-memory lookups are different properties of this design.

Swarm adds a managed control plane

The February Swarm announcement places horizontal scaling in Dragonfly Cloud. It describes growing an individual shard vertically before adding nodes, centralizing cluster metadata in a control plane, and transferring slots through multiple parallel TCP connections. That is a managed scaling design built around the engine's ability to use a larger node.

Centralized metadata changes who coordinates membership and scaling operations. It does not turn multiple machines into one failure domain or remove data movement from a resize. Parallel transfer addresses how that movement proceeds, while vertical growth can reduce how many nodes are needed for a given workload. These are related benefits with separate mechanisms.

I see Swarm as a continuation of the vertical scaling argument rather than a reason to begin with a cluster. A workload that fits the engine on one machine and a workload requiring managed expansion across machines present different operational problems. The cloud offering provides a path between them without making horizontal distribution a prerequisite for using the engine.

Managed operation has its own product scope

The August Cloud Enterprise announcement adds another distinction. It packages capabilities including autoscaling, read replicas, multi-region backups, and direct engineering support. Bring-your-own-cloud is explicitly labeled beta in that announcement, with the managed service running inside the customer's cloud account.

Those features concern how the service is operated and purchased. They should not be inferred from finding a compatible command in the engine or downloading the self-hosted binary. Running a service inside a customer's account also changes its placement without making that service equivalent to operating the engine independently.

The source-available license remains my main caveat. The license included with 1.35.1 is Business Source License 1.1 with an additional use grant. That grant permits use within one's own product or service subject to its restrictions, including limits concerning in-memory datastore offerings and competing services. Community growth and a broader managed product do not alter those terms. A self-hosted deployment still has to fit the permissions of the particular version's license, independently of whether its capacity requirements fit one machine.