Dragonfly combines text and vector rankings

Dragonfly 1.39.0 adds FT.HYBRID, which combines text retrieval with vector similarity search. The release notes also introduce phrase matching, default stemming for text fields and additional relevance scoring. The useful question is how the engine combines evidence from those retrieval methods. Returning a single ordered result requires a policy for scores that begin with different meanings.

I like Dragonfly's ability to use a larger machine without immediately requiring a cluster. Combining retrieval methods inside the server could simplify some application work, but I would want to inspect the ranking policy before relying on it. An application can accept the command syntax and still choose unsuitable weights, candidate counts or language settings. Version 1.39 makes those decisions part of the search design.

Fusion chooses how two result sets contribute

The hybrid implementation exposes LINEAR and RRF combination methods. Its internal result structure keeps separate text scores, vector distances and flags indicating whether a document appears in each input. This allows a result to contribute through one retrieval method even when the other method did not return it. The combined score depends on which evidence exists for that document.

For linear combination, the implementation normalizes text scores by the maximum text score among the collected documents. It transforms vector distance using the minimum and maximum distances in that candidate collection. With a nonzero distance range, the nearest distance maps to one and the farthest maps to zero. An equal-distance collection receives a vector contribution of one for its vector results.

The final calculation multiplies normalized text by ALPHA and normalized vector contribution by BETA, then adds them. A document absent from one input receives zero for that contribution. The patch's comment explicitly assigns alpha to text and beta to vector. I would record that assignment in application configuration instead of relying on the names to explain themselves.

The normalization also means the score is relative to the retrieved candidate set. Changing which documents enter that set can change its maximum text score or vector-distance range. I would avoid interpreting the resulting value as a stable probability or applying an unexplained threshold across unrelated queries. The implementation supplies a ranking calculation, without establishing such a probabilistic interpretation.

Reciprocal-rank fusion uses positions

RRF first orders text results by descending text score and vector results by ascending distance. It assigns ranks beginning at one, with key ordering used to break equal-score or equal-distance ties in the patch. Each available rank contributes the reciprocal of the rank plus a configurable constant. The two available contributions are added for a document present in both inputs.

For example, if the constant is sixty, a first-place appearance contributes 1 / 61. A second-place appearance contributes 1 / 62. These are consequences of the formula in the implementation, not measured relevance improvements. Documents supported by both inputs can accumulate contributions from both lists without requiring their original numerical scores to use the same scale.

RRF therefore avoids selecting a numerical conversion between a text score and a distance. It still requires choices about the candidate population and its constant. I would compare both fusion methods using labelled queries. The relevant outcome is whether the application retrieves and orders useful documents, including cases where text and vector retrieval disagree.

Candidate selection deserves its own test. The parser supports candidate counts and an RRF window, while the returned result count has a separate limit. A document omitted before fusion cannot receive a useful combined ranking afterward. Increasing only the number of returned results does not establish that the earlier retrieval stages considered enough candidates.

The parser also distinguishes KNN retrieval from vector range retrieval. In the patch, a range request combined with FILTER produces an explicit error. I would check the exact form needed by an application before assuming that every filter available elsewhere in search can be attached to this command. The vector-parameter validation separately checks that the supplied bytes decode and that their dimensions match the HNSW index.

Those checks suggest useful negative cases for a client integration. A missing vector parameter should reach the application's error handling, and an incorrectly sized vector should not silently become a different query. I would include these cases alongside relevance tests so that a successful search result is distinguishable from a rejected request in application telemetry.

Text scoring uses aggregate statistics

The release fixes text scoring that depended on shard-local document statistics. The global-statistics patch collects document counts, per-field document counts and total lengths, plus term document counts. It merges the shard contributions before supplying statistics to the scoring phase. This addresses ranking changes caused by how documents happen to be partitioned across engine shards.

The aggregation preserves totals rather than averaging precomputed averages. For a field, the code sums document lengths and document counts, then divides the totals to obtain the average. That is necessary when shards hold unequal populations. Taking an unweighted average of local averages would give a small shard the same influence as a large one.

I consider this important for a system whose scaling model uses internal workers. The application should not need to treat the local distribution of documents as a relevance parameter. The release describes a two-phase statistics pass before search, so I would include that work in latency measurements. A relevance fix can change execution cost even when the query text remains identical.

Version 1.39 includes BM25STD and TFIDF scorer options, with field term frequency and document-length tracking. WITHSCORES and SCORER expose scoring through search commands, while aggregation can add a score field. These facilities make the inputs to ranking more inspectable. I would retain component scores in evaluation output when comparing fusion strategies so that a ranking change has an explanation.

Phrase positions and stemming change matching

Exact phrase queries and approximate phrase matching with slop arrive in the same release. The notes describe per-document token positions stored in compressed posting lists. Matching a phrase needs positional information because the presence of its words alone does not establish that they occur together in the required order. Slop allows a configured departure from exact adjacency.

"hello world"
"hello world"~2

These are query-expression forms documented by the release, not complete client commands. I would test phrase queries with repeated words and separated occurrences in representative documents. Such examples can expose whether a query expresses the intended proximity requirement before any text score is combined with vector evidence.

TEXT fields now apply Porter stemming by default through Snowball/libstemmer. The release supplies a per-field NOSTEM option, an index-level LANGUAGE setting with English as the default, and a per-document LANGUAGE_FIELD. Those choices affect matching and should accompany an index's schema review. I would not leave the default language unexplained in an application containing several languages.

I would also distinguish text fields used for natural-language retrieval from fields containing identifiers whose exact form matters. The existence of NOSTEM gives the index author a specific control. Whether to use it depends on the meaning of the field, and labelled query tests should include the terminology the application expects to preserve.

Lua and protocol fixes remain part of the upgrade

The search work accompanies several security corrections. The release wraps Lua's load() to enforce text-only mode, preventing supplied binary bytecode from taking that loading path. It also hardens rawset, setmetatable and getmetatable around the global environment and library tables. Those changes concern the execution environment of scripts, independently of which search commands an application adopts.

The notes also identify a RESP injection issue in redis.error_reply() and redis.status_reply(). Strings containing carriage-return and newline sequences could inject frames into the response stream. I would include scripts that construct status or error replies in compatibility testing, because their output crosses a protocol-encoding boundary. The security fix deserves consideration even in deployments that never enable hybrid search.

None of this establishes that an operator should accept arbitrary scripts from untrusted callers. It identifies specific corrections in this version. I would preserve the application's existing decision about who can execute scripts and separately check whether legitimate scripts depend on behaviour changed by the hardening.

My evaluation would begin with fixed datasets and labelled queries, recording the text contribution and vector contribution before comparing the combined order. I would vary candidate counts separately from returned limits and make language settings explicit. The server can now calculate the combined ranking, while the application still needs evidence that its chosen fusion policy and matching rules produce useful results.