Dragonfly fixes vector ranking and JSON retrieval semantics

A nearest-neighbour query can return plausible-looking results while ranking the wrong vectors first. Dragonfly's vector-search fix changes cosine scoring from similarity to distance. That one arithmetic change explains a substantial part of the stabilization work in the December 5 release of Dragonfly 1.13.

The release describes the search API as beta and identifies vector and structured search as supported uses. Beta availability was already announced in November, so this release is better understood through its concrete fixes than as the first appearance of search. The changes concern ranking, result fields, and the relationship between stored documents and their query representations.

I find search within Dragonfly interesting because it adds queries over values already held by a Redis-compatible datastore. That can simplify a particular application design. It also gives the engine more consistency obligations than returning bytes for a known key, especially when a result combines filtering with a computed distance.

Similarity and distance order results differently

The release-tagged vector utilities implement Euclidean and cosine distances. Euclidean distance takes the square root of the sum of squared coordinate differences. Cosine distance uses one minus the normalized dot product.

Before the fix, the cosine function returned the normalized dot product directly. That expression describes cosine similarity for nonzero vectors. Turning it into a distance requires the subtraction from one because vectors with similar directions should have a small distance. A nearest-neighbour algorithm using distance ordering cannot substitute a larger-is-more-similar value without changing how it selects results.

For parallel nonzero vectors pointing in the same direction, the normalized dot product is one and the corrected distance is zero. Perpendicular vectors have a normalized dot product of zero and a distance of one. Opposite directions produce a larger distance. These outcomes follow directly from the formula implemented in the source.

The added regression test uses four two-dimensional vectors pointing along opposing axes. It then queries with vectors directed roughly toward each axis and checks the selected result. The query magnitudes differ, which helps demonstrate that the cosine case concerns direction rather than raw coordinate size.

The implementation also has an explicit branch for a zero denominator and returns zero in that case. That is a property of this code, not a general mathematical definition of cosine distance for zero vectors. Applications assigning meaning to degenerate vectors need to account for the actual branch instead of assuming the nonzero-vector formula applies.

Vector bytes need a defined representation

The same utility file converts a byte string into an owned floating-point array. Its comment explains that the incoming memory may not have the alignment required for direct floating-point access. It copies the bytes into allocated float storage instead of simply casting the string pointer.

This is another compatibility boundary hidden beneath an ordinary command argument. A protocol transports bytes, while the vector code needs elements with a particular representation and alignment. An argument that looks like a textual array is not automatically the binary float sequence expected by the vector path.

The tagged search tests construct vector arguments from float bytes. They define dimensions in the schema and pass query data through parameters. That makes dimensionality and argument encoding part of the query contract, alongside the distance selected for the field.

The structure also separates document selection from nearest-neighbour selection. A query can filter documents and then apply a KNN expression to the candidates. The test accompanying the score-alias fix uses a text field filter before its vector expression. The two operations have different roles even when the client submits them as one query string.

Sorting must preserve the vector score

The score-alias fix addresses a query that wraps KNN results in sorting. A vector expression can assign a name to its computed score. Sorting and return-field handling must continue to recognize that name after the query acquires another node around the vector operation.

Previously, the sorting code skipped another sort whenever its child was a KNN node. The fix makes that shortcut conditional on the requested sort field matching the KNN score alias. Sorting by a different field is a different request and must not inherit a shortcut intended for distance ordering.

The patch also carries the score alias through aggregation when the sort field matches it. Without that information, later result construction can lose the computed field even though the nearest-neighbour step found the correct candidates. Correct selection and correct projection of the result are separate requirements.

Its regression test combines a text filter, a KNN expression with an alias, descending sorting, and an explicit return field. The expected result includes the aliased vector score. That combination is useful because testing the vector operation alone would not exercise the wrapper that obscured the metadata.

I regard this as a practical reason to examine composed query behaviour during a beta. Individual operators can work while their combination changes what the response contains. The fix supplies a specific example of information that has to survive transitions between query nodes.

Structured search is distinct from full-text matching

The tagged tests demonstrate text, numeric, and tag fields. Tag values do not participate in ordinary full-text matching in the tag test. A tag query instead names the field and uses its category-selection syntax. The distinction affects how a schema makes the same document bytes searchable. A small hash-based example follows the tested schema and tag-query forms:

redis-cli HSET document:1 color 'red, green'
redis-cli HSET document:2 color blue
redis-cli FT.CREATE colors ON HASH PREFIX 1 document: SCHEMA color TAG
redis-cli FT.SEARCH colors '@color:{red}'

The prefix restricts which hash keys belong to the index. The field type defines how the value is indexed, and the query selects a tag through that declared field. Searching for an unqualified word is a different operation from selecting a tag, even if the visible string happens to be the same.

The tests also index numeric fields and combine range expressions. JSON schemas can identify nested paths and give those paths query aliases. These capabilities explain the release's description of structured search without requiring vector similarity to become the model for every query.

The index introduces additional stored information. The test suite measures index and entry counts as well as estimated memory usage. Search therefore has its own capacity consequences alongside the documents it makes searchable. Retaining the original key-value data does not make the index free. The prefix and schema are consequently capacity decisions as well as query declarations. A narrower prefix selects fewer candidate documents, while field declarations select which parts of those documents need searchable representations. The test cases expose those choices independently, so an index definition can be explained without treating the entire database as one undifferentiated search collection.

The JSON root has two response forms

The JSON retrieval fix separates a legacy dot root from the JSONPath dollar root. Its regression test retrieves the same document through both paths. The dot form returns the compact document itself, while the dollar form returns an array containing that document. The visible difference can be inspected with:

redis-cli JSON.SET sample . '{"name":"cache"}'
redis-cli JSON.GET sample .
redis-cli JSON.GET sample '$'

The final two responses describe the same stored object through different result conventions. A caller decoding the response as an object cannot substitute the array-wrapped form without changing its expectations. Correct path parsing therefore includes result shape, not merely selecting the right underlying value.

The patch represents the dot root with an absent parsed expression and evaluates it as the document itself. Other supplied paths pass through the JSONPath parser. It also centralizes parsing of formatting options and reports argument-parser errors before executing the retrieval operation.

Connection cleanup remains part of datastore correctness

The release notes identify a client-disconnection crash regression. A connection-close patch adds removal of watched-key registrations during close. Those registrations refer to transaction execution information associated with a connection, so their lifetime must end when the connection can no longer own that state.

The patch calls the existing unwatch operation from the close path. This change demonstrates a concrete cleanup requirement without establishing that every disconnect-related failure had the same root cause. The release contains other connection fixes as well, and the announcement describes the overall regression at a broader level.

Version 1.13's stabilization work concerns these observable boundaries. Distance calculations determine ranking, aliases determine returned score fields, and JSON path conventions determine response structure. Connection cleanup separately keeps server-side registrations consistent with the lifetime of the client state that created them.