Dragonfly 0.12 adds defragmentation machinery

Deleting data does not necessarily release every memory page that held it. Dragonfly 0.12.0 adds machinery to move supported allocations off underused pages, alongside continuing replication work. The December 7 release advertises active defragmentation, but the tagged implementation has an important qualification. Its default threshold is deliberately configured to disable normal execution of that task.

The engine-shard source at v0.12.0 places the work in an idle task and checks memory thresholds before beginning. This is incremental maintenance with explicit eligibility rules. It should not be described as automatic compaction of every object in every normal installation.

I consider memory behavior central to the attraction of a cache that scales up within one process. Capacity planning depends on more than the size of the live values. The relationship between allocated objects and committed pages can determine whether removing data produces the expected reduction in process memory.

Underused pages can retain live allocations

The defragmentation patch adds a check for whether an allocation sits on an underutilized allocator page. For a supported string representation, the operation allocates replacement storage, copies the bytes, and frees the original allocation.

The intended effect is to move live data away from sparsely occupied pages. It does not change the logical string value. Freeing an old allocation can help the allocator recover page space, but a relocation is different from deleting a key or evicting a cached value.

This distinction explains why ordinary deletion is insufficient as a description of memory recovery. A page can still hold some live allocations after other objects have been removed. The remaining objects can keep that page relevant even when much of its space is unused.

The original patch also changes the allocator's decommit delay setting. That is related to when freed pages can cease contributing committed storage, while relocation concerns which allocations remain on a page. Both parts matter to understanding the intended memory effect.

Supported representations determine what can move

The compact-object implementation dispatches defragmentation according to the value's representation. The first patch supports allocated string storage while excluding several other forms. A blanket statement that every data structure is compacted would exceed what this implementation does.

Pull request 534 adds the small-string representation before the release. Its code checks the page containing the externally stored portion, allocates a new portion when needed, copies it, frees the previous location, and updates the stored pointer.

That is a separate path because a small string's layout differs from the ordinary allocated string wrapper. The implementation retains an inline prefix and handles the remaining bytes through a segment allocator. Defragmenting it requires working with that representation's address translation and allocation interface.

The distinction also shows why completion cannot be inferred from a function name. A general defragmentation entry point can return without moving anything for an unsupported or irrelevant representation. The effect depends on the actual storage form and the utilization check, rather than applying uniformly to every visited value.

The task has a budget and a cursor

The tagged shard task checks current memory usage against a threshold derived from maxmemory. It also compares committed memory with used memory. The default committed-to-used ratio threshold is 1.3, while the per-page utilization threshold is 0.8.

Those are separate decisions. One asks whether the process-level difference is large enough to justify work. Another asks whether a particular page is underused enough to make an allocation eligible for relocation. They should not be collapsed into one measure of fragmentation.

The memory-usage threshold defaults to 1, with a source comment explicitly identifying that choice as disabling the task. The feature therefore exists without establishing that every default server is actively performing this maintenance. Any account of the release should preserve that operational qualification.

When the task runs, it traverses the default database's primary table with a cursor. Each invocation limits the number of traversals and saves the cursor for continuation. The code checks values for possible relocation and counts attempts separately from successful reallocations.

The task also records invocation counts. A scan that finds no eligible allocation can therefore be distinguished from a scan that moves data. That matters because merely invoking maintenance does not establish that memory has been reclaimed.

The source comments say the fiber runs with exclusive access to its shard. This uses Dragonfly's ownership model to avoid competing access while it changes internal pointers. The maintenance still has a cost in traversal and copying, so bounded work and continuation state are part of the design.

Replication has an initial and an ongoing stage

The December release notes say Dragonfly can connect as a replica behind Redis through REPLICAOF. They separately describe Dragonfly-to-Dragonfly replication as incomplete, with the initial full-synchronization stage working and continued development on stable-state replication.

The tagged replica implementation contains separate paths for an upstream Redis server and an upstream Dragonfly server. It performs synchronization setup, loads initial data, and has distinct stream-consumption machinery for the ongoing phase.

The presence of that machinery should not override the release's explicit maturity statement. Code for transitioning stages and consuming streams is evidence of implementation work. It is not a declaration that native replication is complete or that the release supplies a finished failover system.

I prefer keeping those claims separate because a synchronized initial copy and a maintained replica solve different problems. An initial copy establishes data at a point in the synchronization process. Continuing writes require another mechanism to preserve the relationship afterward. The release calls out that boundary instead of presenting a successful connection as the whole feature.

The December changes consequently make internal progress visible in two areas. Supported allocations can be relocated under explicit maintenance conditions, and replication has identifiable initialization and stream-processing stages. For operators, the relevant qualifications are the default defragmentation threshold and the stated incompleteness of native replication, both of which constrain what this version can be expected to do.