Dragonfly 2.0 extends compatibility through its storage formats

Dragonfly 2.0 extends compatibility into the details of persisted data. A Valkey hash field can carry an expiration timestamp whose binary representation differs from Dragonfly's, and loading it requires more than recognizing the file header. The 2.0 release, published September 16, adds Valkey 9 RDB loading and GEOSEARCHSTORE alongside changes to replication retention and memory handling. Its authors describe the major version as a maturity milestone after three years, while explicitly saying this particular release does not introduce a new major feature by itself.

I value Dragonfly for the operational possibility of scaling one machine before running a cluster. That depends on more than throughput. Persisted records have to retain their meaning during import, replication must have enough history to resume after interruption, and occasional large operations should not dictate permanent buffer allocations. These are concrete parts of the release's production-readiness claim.

Valkey hash expiry needs its own decoder

The Valkey loader patch recognizes the VALKEY file signature and the Valkey format version numbered 80. That internal format number is separate from the product's version name. The loader retains its existing Redis-signature path and explicitly checks supported versions according to the detected signature.

The patch adds object type RDB_TYPE_VALKEY_HASH_WITH_EXPIRY_MS. Like Dragonfly's expiring hash representation, it supplies field, value and expiration information. The field and value use RDB string encoding. Valkey's expiration uses a raw little-endian signed 64-bit integer, while Dragonfly's existing representation encodes that part as an RDB string. Reusing the old decoder unchanged would misinterpret the bytes.

The new ReadValkeyHashExpiry helper reads that integer and rejects values below negative one. Nonnegative timestamps pass through normalization for Dragonfly's field-expiry representation. The calculation adds 999 before dividing milliseconds by 1000, rounding the converted value upward to a second boundary. It then caps the value to the supported internal range and adjusts it relative to the member-expiry base.

This conversion is part of compatibility, not an incidental parsing detail. The imported field has to preserve the distinction between persistence and expiration even though the engines use different encodings and time representations. The patch's regression fixture contains expiring, persistent and overflow-labelled fields. Its assertions inspect the loaded values and use HPEXPIRETIME to check both an expiration timestamp and the persistent-field sentinel.

HPEXPIRETIME hash FIELDS 2 expiring persistent

That command form comes from the loader regression test. It makes field lifetime observable separately from whether the hash loaded successfully. A file parser that reconstructs all values but loses their expiry metadata would satisfy a basic value comparison while changing the dataset's future behaviour.

Backlog retention becomes a time and memory policy

The backlog redesign replaces a fixed-entry approach with age and byte controls. A count of journal records does not express a stable duration of recoverable history. Write rate determines how fast that count is consumed, and record sizes determine how much memory it occupies. Time and bytes expose those resource questions directly.

The final 2.0 source defines shard_repl_backlog_time_ms with a default of 5000. Its description calls this a target retention age and says older entries are evicted on later journal writes. Zero disables time-based eviction. The append path performs cleanup, so reaching an age boundary does not itself require a timer to remove the record at that instant.

An explicit shard_repl_backlog_max_bytes value is the maximum retained bytes for each shard. The default zero selects maxmemory / shard count / 200, allocating a share equivalent to half a percent of configured maximum memory across the shards. An explicit per-shard value therefore needs multiplication by the shard count when estimating the aggregate budget. The initial implementation patch described a different scope, which makes the versioned source important here.

The final release also preserves shard_repl_backlog_len. A nonzero legacy entry limit selects the old retention mode when neither new limit is explicitly configured. That path logs a deprecation warning and disables time-based and byte-based eviction. Explicitly configuring either new setting selects the new mode, including configuration through the corresponding environment flags.

The new controls consequently do not erase an existing deployment's policy merely because it upgrades. A configuration carrying the legacy flag can retain the entry-count behaviour. Conversely, adding a new limit changes which mode the initializer chooses. Understanding that selection is necessary to explain an observed retention window after an upgrade.

The backlog tests distinguish age cleanup from memory eviction. One advances the test clock and confirms that an old entry disappears on the next append. Another limits age-only cleanup to a bounded number of entries per call. Tests also cover a record larger than the configured budget, followed by a later small record that evicts it. These cases show that retention is a policy applied as the journal progresses.

Buffer capacity stops following the largest object forever

The serializer buffer change adds max_rdb_save_serialize_buffer_capacity, defaulting to 4194304 bytes. After building a flushed output blob, the serializer asks its buffer controller to shrink retained capacity to that limit. Assertions require that the internal buffer and any pending prefix are empty before shrinking.

The release describes the previous problem with a large object leaving its large buffer allocated after the work finished. This is retained capacity rather than live dataset size. A process can finish serializing an unusually large value and still carry the associated allocation unless it has an explicit reclamation policy.

The regression test makes the new behaviour precise. It sets an eight-KiB cap, writes data one byte larger than that cap and confirms that capacity grew. After flushing, the serialized length is zero and buffer capacity equals the cap. A subsequent small entry fits and flushes without growing it again. The test checks reusable bounded capacity, rather than requiring every flush to free all storage.

The configured limit applies to what the serializer retains after flushing. It is not a promise that processing a large object never needs a larger buffer. The test deliberately crosses the limit before checking reclamation, and the implementation shrinks only after constructing the output blob. This distinction matters when interpreting a memory graph during a snapshot. Peak allocation during work and the capacity left after that work answer different questions. The new control addresses the latter without making the former disappear from the serialization process.

Other connection changes follow the same concern at a different scale. The release refreshes memory telemetry once per parse cycle instead of after every pipelined command and makes connection-memory introspection constant time through cached mutable-state usage. Those changes reduce the work required to account for connections. They are distinct from reducing the bytes held by each connection.

The notes also describe an optional shared RESP V2 read buffer, disabled by default. Their projected reduction for a large connection population is explicitly future work not realized by that commit. Treating the projection as a measured 2.0 saving would overstate the release. The implemented reclamation and accounting changes already give a concrete explanation of what changed.

RESTORE rejects implausible allocation requests

The RESTORE hardening bounds declared element lengths and collection counts by the remaining supplied input before allocating. The bounded restore source records the payload size. RemainingBytes accounts for buffered input and bytes the source can still provide. A declaration larger than that remaining input fails before it can force a large resize or reservation.

The tests include set, hash and sorted-set payloads declaring huge element lengths, plus compressed data declaring an excessive compressed length. Another case supplies a large declared set-member count with enough real initial data to reach the reservation path. Each must return a bad-data error, leave the destination absent and keep the server responsive to PING.

This is a useful boundary for a compatible importer. Recognizing a format does not mean trusting its allocation claims. The decoder must accept valid data and reject inconsistent declarations before they become memory requests. Version 2.0 separately rejects duplicate hash fields in restore payloads, addressing another case where syntactically readable input has invalid collection semantics.

My view of the major version rests on these explicit behaviours. Valkey field expirations gain a decoder with a documented conversion, backlog history gains selectable retention controls, and serializer buffers gain a retained-capacity limit. The release also adds GEOSEARCHSTORE with shape queries, sorting, limits and journaling. Together these changes extend the set of existing data and command behaviours Dragonfly can support while making its surrounding memory work more bounded.