October 16, 2024·4 min read·engineeringdragonflyhashesexpirationcompatibility

Dragonfly adds expiration updates for existing hash fields

Dragonfly 1.24 adds HEXPIRE and a generic FIELDEXPIRE command. The implementation supports unconditional field expiry, while conditional options remain outside this initial change.

Setting a hash field's value and setting its lifetime are different operations. Dragonfly already has HSETEX for writing fields with an expiry. The HEXPIRE implementation adds a way to update the lifetime of fields that already exist, without making the caller supply their values again. Its tests exercise both ordinary hashes and fields previously written with a lifetime.

Dragonfly 1.24.0, released on October 16, includes HEXPIRE alongside the more generic FIELDEXPIRE. The release describes itself as maintenance-focused, but these commands extend how applications can model independently expiring data inside collections. The initial implementation omits the conditional NX, XX, GT, and LT options discussed in its proposal.

I find field expiry useful because the lifetime of a collection is often different from the lifetime of each item it holds. Dragonfly's ability to keep more work inside one process is attractive to me. A collection operation that expresses the required lifetime directly can also reduce application-side coordination around separate keys.

A field lifetime can change without rewriting its value

The hash-expiry tests first create several fields with HSET. They then assign a ten-second lifetime through HEXPIRE, advance the test clock, and expect the hash contents to be empty. Another test advances only nine seconds and checks that the values remain present. The pair establishes both expiration and the absence of premature removal for that interval.

HSET cache-entry first alpha second beta
HEXPIRE cache-entry 10 FIELDS 2 first second

This follows the tested grammar. FIELDS introduces a count and the field names, so the caller does not pass the existing values. The command returns one result per named field. A successful assignment receives 1, while a missing field receives -2 in the tests.

The per-field result is useful when a request contains both present and absent fields. A single boolean could not express that distinction. An application can inspect the array to determine which targeted fields existed when the operation ran. The missing-key test returns -2 for each requested field, and another test confirms that the expiry command does not create a missing hash field.

The implementation also tests fields created through HSETEX with a longer lifetime. Applying HEXPIRE replaces that lifetime with the shorter one, after which the fields expire at the new deadline. This makes lifetime renewal or shortening independent of value replacement for the supported unconditional case.

The initial syntax omits conditional changes

The proposal explicitly says it does not add the optional conditional flags. That statement matters when describing the command as compatible with another server's field-expiry interface. The shared command name and basic field-list form establish a useful common operation. They do not establish every possible option of that interface.

In particular, an application cannot infer support for changing a lifetime only when it becomes longer or shorter. Those decisions correspond to options the proposal leaves for further work. The unconditional path replaces the lifetime of an existing field according to the supplied duration. The caller's requirements determine whether that path is sufficient.

This is also separate from HSETEX NX, which controls conditional insertion while supplying a value. The existence of an NX option on one command does not imply that another command implements its own conditional rules. Reading the command-specific syntax avoids combining distinct operations into an unsupported request.

Generic member expiry extends beyond hashes

The same patch adds FIELDEXPIRE across hash and set implementations. Its tests use the command on a set's members and on hash fields. Both return per-member results and remove the targeted items after the clock advances. The generic command uses a simpler list of member names without the FIELDS count shown in HEXPIRE.

SADD temporary-members first second
FIELDEXPIRE temporary-members 10 first second

This example follows the set test's form. It shows why FIELDEXPIRE and HEXPIRE should not be described as identical spellings of one protocol operation. One accepts different collection types and has its own argument layout. The other provides the hash-specific interface with an explicit field count.

I regard that distinction as important for portability. A Dragonfly-specific operation may fit an application well, but the decision should remain visible in its command usage. The more general command should not be presented as something every compatible Redis client can assume exists on every server.

Recovery must preserve field records across chunks

A related snapshot-loading fix addresses very large hashes whose fields carry TTLs. The loader represents each element through a key, value, and TTL sequence. Splitting that sequence at an arbitrary chunk boundary could make the next segment begin with a value or TTL where the parser expected a key.

The correction keeps those segments aligned to groups of three elements when TTLs are present. This is a recovery-format issue, separate from the new commands, but it affects the same data model. Per-field lifetimes must survive serialization boundaries as well as work during live updates. The pull request explains the parsing failure in terms of record alignment, rather than a general inability to load large hashes.

BITPOS changes an observable reply

The ACL rule parser also accepts ALLCOMMANDS and NOCOMMANDS. Its tests inspect the resulting ACL listing and expect the equivalent all-command grant or denial. This improves configuration compatibility without introducing a new permission category. Existing account definitions using those spellings can express the same broad command policy through Dragonfly's parser.

The BITPOS correction is explicitly marked as a breaking change because clients can depend on its old response. The tests distinguish searching for zero in a missing key, which returns 0, from searching for one, which returns -1. They also reject bit-mode arguments other than zero or one.

That is a small but concrete compatibility correction. A caller using the old missing-key result can observe a different integer after upgrading, even though the new response follows the intended semantics. Version 1.24 therefore includes both additional expiry operations and corrections to existing replies. For field-expiry users, the new per-field result arrays and the loader's aligned TTL records define the corresponding behaviour during live operations and recovery.