Dragonfly moves expiration metadata into keys

Dragonfly 1.38.0 removes the separate expiration table and embeds expiration metadata in keys. The release announcement reports about 26 percent less memory for its TTL workload, with ten million keys occupying 665 MB against 900 MB before the change. It also warns that changes to core data structures may alter behaviour and asks operators to report regressions around expiry, replication or backups. The storage change deserves attention alongside the new commands.

I consider Dragonfly when scaling one machine can keep a cache simpler to operate. Per-key overhead matters to that choice because small values can leave metadata responsible for a substantial share of memory. A reduction in overhead can change how much useful data fits on the machine. I would still make expiry behaviour part of the acceptance criteria, since an efficient representation has to preserve the application's expectations about when data disappears.

Expiration no longer needs a second table entry

The expiration-table removal describes the previous arrangement. An expiring key required an entry in the prime table and a separate entry in ExpireTable. The author estimates the extra cost at roughly 26 to 35 bytes per key. Removing that second entry addresses overhead that exists independently of the size of the application value.

The prerequisite CompactKey change adds SDS_TTL_TAG. It stores a key string as SDS alongside an int64 TTL and supplies operations to set, clear and retrieve that TTL. Its conversion support covers the existing inline and other key representations. Expiration therefore becomes part of the key's representation, with conversion work when that representation changes.

The server integration replaces expiry-table operations with expiration-time operations on CompactKey. It also changes active expiry scanning to traverse the prime table. That is a meaningful algorithmic consequence of removing a dedicated structure. The server now has to find expiry candidates while traversing a table that also contains keys without expirations.

A follow-up cleanup removes iterator types that existed to pair prime-table and expiry-table positions. Read operations can return a single table iterator, and update structures no longer carry the expiry iterator. The change therefore simplifies interfaces as well as reducing stored entries. It removes state that several code paths previously had to keep in agreement.

The benchmark describes a particular key population

The memory comparison in the removal patch uses ten million generated keys with expirations. Its population command specifies small values and an expiry range. That is a useful case for exposing per-key overhead, but it does not establish the same percentage reduction for every dataset. Large values, keys without expirations and different key representations change the proportion attributable to expiry metadata.

I would estimate the benefit using the number of expiring keys before using total cache size. A cache dominated by many small expiring records has a different opportunity from one dominated by a few large objects. The implementation identifies the overhead being removed, which is enough to guide a representative measurement without assuming the published percentage applies unchanged.

The patch also corrects memory accounting around conversions between inline keys and the TTL-bearing representation. It specifically mentions the inline-key counter and key heap tracking, with a regression test for inline-key accounting. Reported memory needs to follow those transitions if an operator is going to use it for capacity decisions.

For an upgrade evaluation, I would test keys that acquire an expiry after creation and keys whose expiry is cleared. I would also compare a population where almost every key expires with one containing relatively few expiring keys. Scanning the main table makes that mixture relevant. These are validation cases implied by the representation change, not claims that those cases fail in the release.

Count-Min Sketch makes count storage bounded by dimensions

Version 1.38 adds the CMS.* command family with RDB persistence. The release lists initialization by dimensions or probability, increments, queries, merging and information retrieval. The core implementation supplies explicit width and depth, an error-rate constructor, weighted merging and total-count tracking.

The versioned source makes the estimate concrete. It hashes an item, derives one counter position in each row, and adds the increment to those counters. A query reads those positions and returns their minimum. Storage depends on the grid dimensions rather than allocating an independent counter for every distinct item.

Collisions explain why this is an estimate. Different items can contribute to a shared counter, so a counter may include increments belonging to other items. Taking the minimum across rows limits the effect of those collisions without reconstructing a separate exact history. I would use such a structure only where the application can accept estimated frequencies and has an explicit reason to bound memory.

The source requires matching width and depth when merging sketches. If those dimensions differ, MergeFrom returns false. It otherwise combines corresponding counters with the supplied weight and updates the total count. Applications planning to combine independently maintained sketches should treat dimensions as shared configuration. Successful initialization at each producer does not by itself establish that the resulting structures can be merged.

The probability-based constructor computes width from the error parameter and depth from the probability parameter. I would document those choices with the application configuration and preserve them in tests. They influence the amount of allocated storage and the interpretation of the estimates. The availability of persistence does not remove the need to choose appropriate dimensions.

Top-K retains candidate identities

The new TOPK.* family tracks heavy hitters and also has RDB persistence. Its initial implementation explicitly separates this public data structure from Dragonfly's internal hot-key tracker. Supporting the public API requires allocator integration and serialization that are not requirements of the internal tracker in the same form.

The 1.38 header gives a more specific algorithm description than the feature name. It describes a counter grid coupled to a min-heap and a decay heuristic. It also says this is not a strict HeavyKeeper implementation because the cells contain counters without the fingerprint-and-count pairs used by the original algorithm. I would preserve that distinction when explaining its behaviour.

The heap retains the identities of candidate heavy hitters, which a frequency query over a Count-Min Sketch does not enumerate by itself. The public Top-K interface can report an item displaced by an insertion and list current candidates ordered by estimated frequency. This makes it useful for an application asking which items are frequent, while CMS addresses estimated counts for items supplied to a query.

Persistence changes how these structures fit an application. The release can save and reload them rather than requiring every process restart to begin with an empty estimate. I would still decide whether the application wants accumulated history or periodic replacement. The existence of a persistent sketch does not define the observation period for a popularity calculation.

Vector range queries express a distance condition

The release also adds VECTOR_RANGE for FLAT and HNSW indexes, including aggregation support. The FLAT implementation introduces a query with a radius and a supplied vector, with an optional distance alias. Its documented query form is the following.

@field:[VECTOR_RANGE $radius $vec]=>{$YIELD_DISTANCE_AS: distance}

A radius condition expresses a different application requirement from requesting a fixed number of nearest neighbours. I would use it when the decision depends on whether a candidate is sufficiently close. The radius still needs to match the application's vector representation and chosen distance measure. Copying a threshold from an unrelated embedding workload would not establish a meaningful acceptance rule.

The HNSW addition introduces range exploration and obtains matching documents through the global registry before fetching their fields from owning shards. Its description includes a small overscan intended to improve recall. I would therefore test range-query recall as part of the application's retrieval requirements, especially when choosing between FLAT and HNSW execution.

My first evaluation of 1.38 would concentrate on the cache's expiring-key population and the behaviour of expiry under that mixture. I would assess sketches with explicit dimensions and observation periods, then test any vector radius against labelled application examples. The core storage change justifies measuring memory again, while its changed scan path justifies repeating expiry and recovery checks on representative data.