Dragonfly reduces duplicate JSON keys and vector storage

Dragonfly 1.37.0 reduces memory use by changing what the engine stores more than once. JSON objects can share their key strings, and HNSW indexes can reference vector data already held in hash documents. The release notes report significant JSON memory reductions and 40 percent lower vector-search memory use with hash documents. Those claims describe different mechanisms, and I would evaluate them separately before translating either into a capacity estimate.

My interest in Dragonfly is the possibility of keeping a growing cache within one multithreaded process. That makes memory efficiency a practical part of the scaling decision. More usable memory can postpone a larger machine or a more complicated deployment. I also want to understand the CPU and ownership costs behind that efficiency. Version 1.37 supplies enough implementation detail to ask more precise questions than whether the new release is smaller.

JSON field names can share storage

The JSON key change connects reference-counted strings to jsoncons for use as object keys. Its description identifies an interned string pool and explicitly adds pool cleanup before destroying a shard's local memory resource. Shared strings therefore have a managed lifetime that must end before their allocator disappears.

Repeated field names are the obvious reason to want this arrangement. If many documents contain the same property names, retaining independent copies of those names spends memory on repeated schema vocabulary. Interning provides shared storage for matching strings. The saving depends on the documents' repetition and the overhead of the pool itself. I would expect a collection with stable field names to be a more relevant evaluation case than a claim based only on total document bytes.

The pull request includes a comparison using a scale-test script with random payloads. In that test, the reported memory use falls from 6.25 GiB to 4.5 GiB. The author describes this as close to a 28 percent reduction. The same comparison takes 203.48 seconds with reference-counted strings against 184.37 seconds without them. Its reported throughput decreases accordingly. These are maintainer measurements for the described test, not numbers from my own systems.

That combination is useful evidence because it shows a resource tradeoff. Lower memory use does not establish that writes become faster. I would compare the cost of a larger memory allocation with any measured throughput loss under my workload. A cache near its memory limit may benefit from a different balance than a service already limited by write processing.

The discussion also describes a random-JSON population test in which reported memory use and resident memory diverge. The author attributes a retained gap tentatively to the hash set used by the pool. A subsequent update says FLUSHALL now shrinks the pool and memory usage falls as expected. That history gives a concrete reason to examine resident memory after deleting data, in addition to measuring allocation while the dataset is full.

I would include a grow-and-delete cycle in an evaluation of this release. Populate representative objects, observe the steady state, remove them, and compare the resulting process memory with the starting point. This is a proposed validation procedure. The implementation discussion supplies the reason for it, since shared string storage introduces a pool whose capacity has its own behaviour.

Hash vectors stop requiring the same copy

The vector work starts with hnswlib support for pointers to vector data. It separates vector data into its own memory block so the library can support references as well as copied data. A server integration then passes a pointer to existing data for hash-backed documents and avoids constructing a temporary vector during index insertion.

JSON documents retain a copy in the implementation described by those changes. The storage representation therefore matters to the reported saving. A vector index over hashes and a vector index over JSON are not interchangeable capacity cases merely because they use the same distance metric or vector dimensions. The release note's 40 percent figure explicitly names hash documents.

Referencing data also makes its lifetime relevant to the index. A separate patch in the release, the defragmentation adjustment, omits key defragmentation when vector data is used as a reference. Even its title explains an important constraint of this approach. Moving referenced storage requires coordination with the structures holding its address. Eliminating a copy transfers some responsibility to storage ownership and movement.

I would assess this change with updates and deletions as well as initial loading. A successful initial index build does not exercise the entire lifetime of a referenced vector. The practical question is whether the application's normal mutation pattern retains the expected memory and retrieval behaviour while documents change. Nothing in the percentage claim answers that question on behalf of a particular application.

The two memory changes also affect different parts of the dataset. JSON key interning reduces repeated object-key storage. Hash vector references avoid duplicating an existing vector representation in the index. I would not add their percentages together or apply both to the whole process. Capacity planning needs a breakdown of which document types and indexes actually occupy memory.

SORT compatibility has an isolation detail

Version 1.37 also supports the BY and GET options for SORT. The BY implementation accepts a pattern used to obtain external sort weights. Its description states that the BY step does not respect transaction boundaries and performs reads at read-uncommitted isolation. It compares the transactional semantics to SCAN and says the ordinary strong guarantees apply when BY is absent.

That qualification belongs next to the compatibility claim. Supporting the syntax does not tell an application whether several external weights represent one consistent observation. If those weights change concurrently, I would establish whether the resulting ordering is acceptable for the use case. A display sorted by approximate popularity and a decision requiring a consistent set of values have different requirements.

The GET addition retrieves external values by substituting each element into a pattern. It also supports # for the element itself and multiple GET patterns. The following command is the form shown by the implementation description.

SORT mylist BY weight_* GET # GET obj_*

Here the weights and the returned associated objects are separate external lookups. I would use an example like this in compatibility tests because it exercises the reason an application uses the options. A test limited to sorting the elements themselves would miss both the projection behaviour and the external-read qualification documented for BY.

Conditional deletion moves a check into the command

The DIGEST and DELEX implementation adds conditional operations on string values. DIGEST key returns an XXH3 digest represented as a 16-character hexadecimal string. Its description gives linear complexity in the stored string's length. It returns nil for a missing key and a type error for a non-string value.

DELEX can compare the current value directly or compare its digest before deleting the key. IFEQ deletes on value equality, while IFNE deletes on inequality. IFDEQ and IFDNE provide the corresponding digest comparisons. The implementation describes atomic execution through the transaction framework and journaling for replication. With no condition, the command delegates to ordinary deletion.

DELEX cache-entry IFEQ expected-value

The example expresses a deletion that depends on the value still being the expected string when the command executes. That avoids expressing the condition as a client-side read followed by an unconditional delete. I would choose direct value comparison where the value is small and its exact identity matters. Digest comparison introduces a distinct representation and still requires the server to hash the stored value.

The pull request records tests against Redis 8.4.0, but I would keep application validation narrower than its broad compatibility language. Exercise missing keys, mismatched values and invalid arguments with the client that will send the command. The command returns a deletion count, so the application must also handle a condition that legitimately leaves the key in place.

For my scaling decisions, the memory work is the main reason to investigate 1.37. I would measure JSON collections with repeated keys separately from hash-backed vector indexes, including the write cost and memory retained after deletion. Any adoption of the expanded SORT options would also carry an explicit decision about the documented external-read isolation semantics.