December 22, 2024·4 min read·engineeringdragonflystreamssnapshotsprometheus

Dragonfly distinguishes attempted and successful stream consumption

Dragonfly 1.26 improves stream format compatibility and consumer activity reporting. It also fixes the metrics response header required by Prometheus 3.0 and separates cloud paths from local directories.

A stream consumer can keep polling without receiving any messages. Treating every attempt as successful activity hides that distinction from an operator. Dragonfly's consumer active-time change stores both the last attempt and the last successful read or claim. The resulting introspection can distinguish a consumer that recently asked for work from one that recently obtained it.

This is part of Dragonfly 1.26.0, published on December 22. The release describes a series of stream improvements towards Redis and Valkey 7.2 compatibility. Those changes include loading additional stream snapshot formats and preserving more consumer metadata, alongside corrections to stream IDs and memory measurement.

I find this a useful measure of compatibility because stream state contains more than message payloads. A replacement engine has to preserve the information applications and operators use to understand consumers. Dragonfly's single-process scaling model remains attractive to me, but a familiar command interface also needs meaningful recovery and introspection behaviour.

A recent poll does not imply recent work

The activity test creates a consumer group on an empty stream and performs a group read. After advancing the clock, it expects an idle duration since that attempt and an inactive value of -1. The consumer exists, but it has not yet completed a successful read.

The test then adds a message and reads it through the same consumer. Both durations advance from that successful operation. Later, another read returns no messages. The idle duration resets to zero because the consumer has just attempted a read, while the inactive duration continues increasing because no new work was obtained.

That distinction is directly useful when interpreting consumer health. A polling loop can demonstrate that a connection and application are still making requests, while successful consumption answers a different question about progress. The new fields let a caller observe both without interpreting one timestamp as evidence for both conditions.

The patch extends consumer information replies and full stream information. Its consumer structure documents seen_time as attempted activity and active_time as successful reading or claiming. This is a semantic change to the stored metadata as well as an extra output field. A client decoding positional reply contents has to account for the additional entries shown in the updated tests.

XINFO CONSUMERS mystream mygroup
XINFO STREAM mystream FULL

These are the introspection forms exercised by the patch. The first reports consumer-level durations, while the full stream form includes the consumer timestamps within the group information. The test checks the timestamp values before and after serialization, linking the live observation to its recovery representation.

The reader accepts formats before the writer changes defaults

The stream-loading correction supports the original stream-listpack RDB type and its second and third variants. Its description explicitly separates accepting those formats from changing the output format. That sequencing matters for deployments in which different engine versions must read the same serialized state.

The subsequent active-time change adds writing of the third variant when stream_rdb_encode_v2 is enabled. The version-pinned serializer defines that flag as false by default. Its description associates the newer form with Redis 7.2 and Dragonfly 1.26, while the older form retains Redis 6 compatibility.

This means loading newer streams and emitting newer streams are separate capabilities in the release. Accepting a newer input does not imply that every snapshot now uses it. The option allows the serializer's output choice to remain explicit. The active-time test enables the newer encoding before dumping and restoring the stream, then checks that the activity timestamps survive.

I regard that separation as a sensible compatibility choice. Expanding what a server can read is different from requiring every other reader to understand a changed output format. The source makes that distinction visible through both the flag default and the recovery test.

Stream IDs and memory measurements also change

The stream timing and sequence correction uses transaction time in stream code and checks for sequence overflow when a request supplies a very large sequence identifier. Its description adds a test for that boundary. The change aligns stream timing with the transaction context already used elsewhere in the engine.

A separate stream improvement patch validates XSETID against the stream's maximum deleted ID. It also adds precise stream memory measurement for MEMORY USAGE and compacts listpack nodes after creating new nodes. These corrections concern the bookkeeping around the stream's logical entries and their physical representation.

The distinction matters when assessing the release's compatibility claim. Loading a message collection is insufficient if ID validation or consumer metadata differs from what the application expects. The cited changes address several specific parts of that behaviour. They support a more concrete account than an unqualified claim that every stream operation is now identical across engines.

Metrics require a response format declaration

The release also fixes Prometheus 3.0 scraping. The reported failure concerns a missing Content-Type header, rather than an unavailable metrics body. Prometheus rejects the target because it cannot determine the scrape protocol from that response under the reported configuration.

The correction sets the metrics response MIME type to text before writing its contents. It also updates the local monitoring stack to Prometheus 3.0. The failure illustrates a compatibility requirement at the HTTP boundary. Returning the expected metric lines does not eliminate the need to declare their representation.

Cloud paths must not create local directories

GCS save/load support already appears in the 1.25 release, so it is not wholly new in December. Version 1.26 includes a cloud-directory correction that stops treating a cloud --dir value as a path to create locally. The author describes unused local directories and permission failures when that creation was attempted.

A storage URL identifies a remote destination, while a filesystem directory requires local creation and permissions. Sharing a configuration field does not make those operations equivalent. The fix keeps local-directory handling out of the cloud path, which removes an unrelated filesystem dependency from snapshot configuration using object storage.