Dragonfly adds blocking stream reads
Dragonfly 1.4 expands Streams support with XREAD, XINFO, and XTRIM. Its blocking read path shows how stream positions and internal shard ownership affect compatibility.
A stream reader needs to distinguish data that already exists from data that arrives after the read begins. That distinction becomes a scheduling problem when a datastore divides keys among worker threads. Dragonfly's June 18 release adds stream-reading and trimming capabilities, including blocking XREAD, alongside fixes aimed at BullMQ compatibility.
The interesting part is the relationship between a stream position and a blocked transaction. The release-tagged implementation first resolves positions and checks for available entries. It waits only when the requested data is not already available and the command permits blocking. That sequence avoids treating every stream read as a request to wait for a new event.
I am interested in these details because Dragonfly's multithreaded design is the reason to consider it for a growing cache. Queue libraries exercise more coordination behaviour than independent key lookups. Their compatibility depends on how commands suspend and resume, as well as on whether the server recognizes their names.
A position defines which entries qualify
The new parser requires STREAMS followed by an equal number of keys and positions. The keys come first, then the corresponding IDs. A read of two streams therefore has two positions, each associated with its own stream. An unbalanced list produces a syntax error before the transaction attempts to retrieve entries.
For an explicit ID, the parser increments the internal starting position. The read should return entries greater than the supplied ID, so the implementation adjusts the lower bound used by its range operation. The externally supplied position describes what the caller has already passed. A short example uses explicit IDs to make the sequence visible:
redis-cli XADD events 1-0 kind created
redis-cli XADD events 2-0 kind updated
redis-cli XREAD STREAMS events 1-0
The final command requests entries after 1-0. This makes an explicit position suitable for continuing a reader after its last processed record. The server can select the next range using the supplied position without remembering a separate cursor for this plain XREAD call.
The special $ position has different meaning. The implementation resolves it to the stream's current last ID, increments that bound, and waits for later entries when blocking is requested. Substituting $ for a previously returned ID changes the question. It asks for entries after the current end, rather than continuing from the position the application retained.
Waiting belongs to the transaction
The blocking form adds a waiting deadline to the position request. This command starts after the current end of the stream and permits the server to wait for a later entry:
redis-cli XREAD BLOCK 5000 STREAMS events '$'
The shell quotes prevent interpretation of the special position. The timeout is in milliseconds in the implementation. A value of zero selects an unbounded deadline, while a positive value computes a deadline from the steady clock.
Before waiting, Dragonfly schedules the transaction and obtains the last IDs of its streams from their owning shards. If a stream already has a qualifying entry, the command takes the read path immediately. This keeps historical reads and waits for future entries within one command while giving them different execution paths.
When no data qualifies, the blocking helper uses WaitOnWatch for the relevant keys. If the command lacks BLOCK, it closes the transaction and returns a null array. The same immediate-return branch applies inside a multi-command transaction. A caller cannot assume that adding a blocking option will make every execution context wait.
The wake-up path has a deliberate restriction. Its comment says that only the shard containing the key that woke the transaction blocks for that transaction to proceed. Consequently, the helper reads the awakened key through a range operation instead of reusing the general multi-stream read operation.
That distinction explains why the implementation has separate paths that appear to answer similar questions. A nonblocking read can collect results across participating shards. A transaction resumed by one key has a different scheduling relationship with the other shards. Reusing the broad read helper would overlook that relationship.
Responses preserve the command's ordering
For an immediately resolvable multi-stream read, Dragonfly collects per-shard results and maps them back to the original argument order. Internal ownership is not the order the client supplied. The transaction exposes reverse argument mapping so that the response can preserve the client's association between stream names and records.
Empty streams do not occupy response entries. The command counts streams with records and emits only those streams, each with its returned IDs and field-value pairs. That response construction is part of compatibility because a library parses the nested reply, not the internal containers used to gather it.
The release includes reply-aggregation fixes for stream commands. These changes fit the same concern as reverse argument mapping. A server can retrieve the correct records and still present a reply that a client cannot consume correctly if nesting or collection construction is wrong.
I regard this as a useful example of the real work in protocol compatibility. The engine has to turn per-shard execution into the exact externally observable command result. That includes the shape of replies and the rules for omitting streams with no qualifying records.
Reading and retaining data are separate operations
The same release adds XTRIM. Its handler parses trimming options, executes on the owning shard, and returns the number of removed entries. A missing stream returns zero in the trimming implementation. The parser recognizes length-based and minimum-ID strategies, so retention can be expressed independently from reading. For example, a length-based trim can use:
redis-cli XTRIM events MAXLEN 1000
This operation changes which entries remain available. It does not advance the explicit ID retained by an application. A reader's position and the stream's retention policy therefore need separate representations in the application model.
The tagged registration lists XREAD, but it does not register XREADGROUP. The parser explicitly rejects the special > position associated with group reads. Version 1.4 expands stream support without establishing complete consumer-group command coverage.
The release's BullMQ work includes broader fixes, and the announcement describes improved compatibility. The stream implementation supports that narrower claim. Blocking reads, reply construction, and trimming now cover additional library interactions, while the absent group-read handler remains a concrete boundary in this version.