Redis 5 Streams separates stored history from delivery state

Receiving a message does not establish that its work has finished. Redis 5 gives that distinction an explicit representation through Streams consumer groups. The stream stores ordered entries. A group records how far it has delivered new entries and which deliveries still await acknowledgement. Application processing happens outside both records.

The October 17 release entry announces the first stable Redis 5 release. The version-pinned notes identify Streams as its principal new user-facing data type and describe fixes to AOF and replication made during development. Understanding what Redis persists is therefore part of understanding the feature itself.

I like Redis when an application can express its state through concrete operations. Streams provides a useful vocabulary for recorded events and unfinished deliveries. The distinction is more valuable than a broad claim that a database has acquired messaging, because it explains what a consumer can recover after it stops.

Entries remain available to independent readers

An entry consists of field-value pairs and an ordered ID. XADD appends it, while range reads can retrieve it without removing it. The stable command implementation supports ordinary XREAD requests using a last-seen ID. Each reader can carry its own position between requests.

XADD orders 1-0 order 42 action created
XADD orders 2-0 order 43 action created
XRANGE orders - +
XREAD COUNT 10 STREAMS orders 0-0

These commands assume a new orders key. Explicit IDs make the example repeatable to read, but they are illustrative identifiers rather than production timestamps. Each new explicit ID must exceed the stream last ID. Automatic ID generation uses * when the application wants Redis to assign one.

The ordinary reader supplies its own progress. Reading an entry does not prevent a second independent reader from retrieving it. This is suitable when several applications need different views of the same recorded entries. The stream contains the data, while each application decides where to resume.

Consumer groups add server-side coordination among named consumers. A group has its own progress and pending-delivery records, so it is a separate processing view of the stream. Different groups need not share the same progress. Within a group, readers asking for new messages use the group position to distribute those deliveries.

Group progress measures delivery

The group structure calls its last_id the last delivered ID and explicitly distinguishes it from acknowledgement. That field determines where new deliveries start. It is not a high-water mark proving that every preceding item has completed processing.

XGROUP CREATE orders billing 0-0
XREADGROUP GROUP billing worker-a COUNT 1 STREAMS orders >
XPENDING orders billing

After the earlier append commands, this creates a group starting before the example entries and delivers one new item to worker-a. The > argument asks for entries not yet delivered through the group new-message position. XPENDING then exposes outstanding delivery information.

The starting ID matters when creating the group. Using 0-0 allows the group to begin with existing entries after that position. Using $ initializes the group at the current stream last ID, so earlier stored entries do not become new deliveries for that group. These are different choices even though the same underlying stream remains available.

Redis 5 also supports MKSTREAM with group creation. That option can create an empty stream when the key does not exist. It removes a setup dependency for a consumer that must establish its group before a producer appends the first entry. It does not make group creation idempotent, since creating a group whose name already exists returns an error.

A pending record has an owner

The group pending-entry list records deliveries awaiting acknowledgement. Redis also maintains a pending list for each consumer, with references to the same delivery records. Those records include the consumer, delivery time, and delivery count. A pending entry is consequently more information than a single group offset can express.

Acknowledgement removes pending references through XACK. In the example, acknowledging 1-0 looks like this:

XACK orders billing 1-0
XRANGE orders 1-0 1-0

The range read can still retrieve that entry because acknowledgement does not delete it from the stream. The operation changes the group processing record. Other groups and ordinary readers can continue to use the stored entry according to their own positions and the stream retention policy.

A consumer can read its pending history by supplying a concrete ID to XREADGROUP instead of >. This has different semantics from asking for new work. The distinction allows a consumer to address deliveries already assigned to its own identity. A read of pending history can also update delivery bookkeeping, since returning an entry is another delivery attempt.

I find that separation useful because it avoids treating receipt and completion as the same event. The application can acknowledge after its work finishes. Redis can retain a record of the unacknowledged delivery without needing to understand what the application was doing with it.

Recovery changes ownership without proving failure

XCLAIM transfers a pending delivery to another consumer subject to an idle-time condition. For example, a recovery worker can request an entry that has remained pending for at least a chosen interval:

XCLAIM orders billing worker-b 60000 1-0

This command is a separate recovery example for the case where 1-0 remains pending. If the earlier XACK has already removed its pending record, an ordinary claim does not recreate that record. The minimum idle argument is in milliseconds, and the implementation compares it with the pending delivery time before transferring ownership.

Idleness is evidence that a delivery has not recently been recorded, not proof that the former consumer has stopped every external action. A worker can pause after completing an external operation and before acknowledging it. Another worker can later claim the pending entry and perform the operation again. This follows from the separate application-action and acknowledgement steps.

An application that cannot tolerate that repeated action needs its own way to recognize completion or make the action repeatable. The stream ID can identify the entry, but Redis cannot make an external side effect atomic merely by returning that ID. No command in this sequence joins an arbitrary external system transaction with XACK.

NOACK changes the bookkeeping choice by suppressing pending tracking for a group delivery. That can suit a use where the application does not need Redis to retain outstanding-delivery state. It also removes the corresponding pending record that ordinary recovery would consult. The option should be understood as a change to the processing contract.

Retention and acknowledgement are independent

XDEL removes specified entries, and XTRIM MAXLEN limits retained stream length. Approximate trimming with ~ permits removal of whole internal packed groups and can leave the stream longer than the requested length. These commands concern stored data, not whether every group has finished processing it.

The consumer-history implementation explicitly handles a pending record whose stream payload is gone. It can return the ID with a null fields result. The pending delivery and the stored entry therefore have independent lifetimes. A pending count by itself does not establish that every corresponding payload remains available for recovery.

This affects the meaning of a retention target. If a group is allowed to lag beyond the retained history, its pending bookkeeping cannot restore deleted data. Choosing the retained length is consequently also a decision about how much delayed processing and replay the application can support. The data structure supplies the operations, while the application defines the acceptable interval.

Persistence must reconstruct more than the payload

The AOF rewrite implementation shows the state Redis needs to preserve. It writes XADD operations for retained entries using their existing IDs. It then writes XSETID to preserve the stream last ID, which may differ from the greatest surviving entry after deletions.

For each consumer group, the rewrite emits group creation with its recorded last-delivered ID. It then reconstructs pending deliveries using claim operations for consumers that have them. A file containing only the visible field-value records would omit the processing state that distinguishes consumer groups from independent reads.

An empty stream is also meaningful state. The rewrite creates an empty stream and restores its last ID, rather than discarding the key simply because it contains no retained entries. Append history and current length are separate facts, and ID monotonicity depends on preserving the appropriate history metadata.

Redis 5 ships these mechanisms as part of the stable release. The launch notes say Streams fixes continued until shortly before release and invite bug reports. The substantive model is already clear in the implementation, stored history has identities, groups track delivery, and acknowledgement updates pending state independently of data retention.