Dragonfly shares its HNSW index across engine shards

Dragonfly 1.36.0 introduces a global HNSW vector index. The release announcement also lists Unicode tag support, JSON defragmentation improvements and django-cacheops compatibility fixes. The vector change interests me because it modifies the relationship between search and the engine's internal partitioning. A shared index introduces coordination into a system whose appeal, for me, is its ability to use a large machine through a multithreaded, shared-nothing core.

My preference for Dragonfly starts with operational simplicity. I reach for it when a cache has outgrown one Redis process and scaling a machine up looks simpler than operating a cluster. Search adds another consideration to that decision. Distributing keys among workers does not automatically produce the search structure an application needs. Version 1.36 makes that distinction explicit by giving HNSW its own process-wide registry while retaining shard-local document storage.

What global means here

The implementation change introduces GlobalHnswIndexRegistry, a singleton that creates, looks up and removes HNSW indexes. Its registry key combines an index name with a field name. Each indexed HNSW vector field therefore has a shared search structure that the server can find independently of the engine shard holding a document. The term global describes this internal arrangement. It does not establish a distributed index spanning arbitrary server instances.

Document identifiers explain how that arrangement works. The change retains a 32-bit local document identifier and introduces a 64-bit global identifier. It places the shard identifier above the local identifier's bits. In the patch, construction is equivalent to this expression, with the return type declared as uint64_t.

return (uint64_t(shard_id) << 32) | local_doc_id;

The server can decompose the resulting identifier into its shard and local document components. That gives the graph an unambiguous document reference while preserving the information needed to fetch the document from its owner. The accompanying hnswlib change widens the library's internal identifier to accommodate this representation. This is a concrete compatibility requirement inside the engine, even though a client continues to work with document keys.

The registry also distinguishes graph ownership from document ownership. Creating an index creates the applicable global HNSW structures and asks each shard to rebuild its contribution. Dropping an index removes the corresponding registry entries. The patch adds explicit shutdown cleanup as well. These operations matter because a shared graph needs a lifetime that remains consistent with the local indexes referring to it.

Query execution crosses the boundary twice

The search path in the same patch treats an HNSW nearest-neighbour component separately from the rest of a query. The server looks for the relevant global index and extracts the KNN node. If the query contains a prefilter, shard-local searching first produces the eligible documents. The server converts their local identifiers into global identifiers before invoking the graph search.

That prefilter path carries a lookup from global identifiers to already serialized documents. When the graph returns candidates, the server can attach the calculated distance to those existing document results. The patch returns early in this case because it already has the document data. This avoids a redundant fetch in the execution path shown by the change.

An unfiltered KNN query follows a different route. The graph first returns distances and global identifiers. The server decomposes those identifiers, groups the required documents by shard, and schedules work on the owning shards to serialize them. It then collects the serialized documents into the search results. A shared graph consequently does not require all document fields to move into a shared document store.

This separation helps explain where I would look for costs during evaluation. An unfiltered search must retrieve the selected documents after graph traversal. A filtered search has already paid for producing eligible documents before the graph search begins. I would test both shapes using the application's actual filter selectivity and returned fields. A single headline query-throughput result would leave those execution differences unexplained.

The patch includes a small regression test with a tag field and a one-dimensional vector field. Three documents have positions of one, two and three. Only the first and third carry the selected tag value. A nearest-neighbour query centred on two must therefore return the eligible documents rather than the document at the exact query position. The example makes the interaction between document eligibility and vector ranking concrete without implying anything about production relevance quality.

Shared indexing has synchronization costs

The implementation adds synchronization around the shared structures. Registry operations use a shared mutex for lookup and exclusive access for mutations. Within the HNSW adapter, a separate mutex protects resizing against insertion. Its resize path checks whether capacity is exhausted before taking the writer lock and checks again after obtaining that lock. Another insertion may have changed the situation while the caller waited.

I consider those details part of the design tradeoff. Sharing a graph changes which operations can proceed independently. The patch explicitly notes that resizing is not thread-safe with insertion, so the code cannot rely on the engine's document partitioning to provide all the necessary isolation. Any assessment of the new arrangement should include index growth and concurrent writes alongside steady-state queries.

The tests also distinguish exact distance ordering from approximate HNSW search. An integration-test adjustment asks HNSW for all points before applying the result limit when comparing against an exact calculation. The test comment describes HNSW as approximate greedy search. I would keep that distinction in application acceptance tests. A correct index implementation and acceptable retrieval quality require separate checks, especially when the application applies restrictive filters.

Nothing in the release announcement supplies a workload-independent speedup for this change. I would avoid converting the architectural change into a numerical claim without measurement. Its immediate significance is that the graph can represent documents across internal shards through one identifier space and one registry-managed index. Whether that improves an application's performance depends on the work its queries and updates require.

JSON defragmentation becomes selective

The JSON work addresses a separate consequence of keeping structured data in memory. The defragmentation implementation replaces a serialize-deserialize round trip over the full object with a tree walk. The new code examines individual nodes and defragments them when needed. It uses knowledge of node storage and the jsoncons library's move operations to perform reallocations and cleanup.

The original pull request deliberately added the implementation without connecting it to the compact object path. The follow-up integration makes the tree walk active and removes obsolete helper functions. It also introduces disable_json_defragmentation, which defaults to false. Setting it to true skips defragmentation of JSON objects. That flag is an escape mechanism for this particular maintenance operation, not a general instruction to disable memory management.

The mechanism matters more to me than an unsupported percentage improvement. A full serialization round trip reconstructs the whole object even when only part of its storage needs attention. A selective traversal can make decisions at the node being examined. The first patch adds benchmarks that compare serialization-based defragmentation with the new approach, including a case where not every node is fragmented. That is a useful test distinction because uniformly fragmented objects do not represent every possible maintenance workload.

I would evaluate the change with JSON mutation patterns that resemble the application. Repeated replacement of large documents and small changes within existing documents can present different allocation behaviour. The release notes establish that the implementation changed. They do not establish that every document shape will save the same memory or consume the same maintenance CPU. Those remain workload questions for an operator.

Compatibility includes application behaviour

The release also contains Unicode tag support. The patch description states that it supports Unicode tag values. For an application that searches multilingual metadata, I would include representative tag values in the upgrade checks. I would avoid inferring broader text-processing guarantees from this specific change, since tag support and the behaviour of other search field types are separate questions.

The django-cacheops fixes deserve the same application-level attention. The first compatibility patch addresses handling of a Django cache script, and a follow-up closes further compatibility issues. These are reminders to test the actual client library and its scripts during a replacement exercise. Successful basic commands do not by themselves verify the behaviour of a cache integration.

My interest in 1.36 is therefore practical. I would compare filtered and unfiltered vector queries while the index grows, and separately inspect JSON memory behaviour under representative writes. The new shared graph creates a specific coordination boundary that deserves measurement. The JSON change gives maintenance code a more selective way to move allocations, with a documented flag available if that path needs to be isolated during investigation.