Dragonfly adds Cuckoo filters and replica acknowledgement waits

Dragonfly 1.40.0 adds Cuckoo filters with persistence and introduces WAIT for replica acknowledgements. The release notes also describe native-width vector storage and additional HNSW controls. These additions address different resource decisions. Filters retain approximate membership information, acknowledgement waits expose replication progress, and vector storage follows the declared representation more closely.

I assess Dragonfly as a way to scale a cache within one multithreaded process before operating a cluster. That makes memory representation relevant, but it also makes command semantics worth reading in detail. A compact filter must suit the application's error model. A replica acknowledgement must support the application's decision after a write. The command names alone do not settle either requirement.

Cuckoo filters store fingerprints in candidate buckets

The versioned Cuckoo filter interface exposes fingerprints and two candidate bucket indices for a hashed item. When both buckets are full, insertion can evict a fingerprint and attempt to move it to its alternate bucket. The implementation limits this relocation process and rolls back swaps if it cannot place the item.

This is a compact representation of membership rather than an exact set of original values. The interface documents possible false positives for membership checks. Its count operation counts matching fingerprints across candidate buckets and subfilters, so collisions can also inflate occurrence counts. An authoritative check is necessary before treating the result as proof that a particular original value exists.

Duplicate insertion has explicit behaviour. Ordinary insertion can add repeated occurrences, with each successful insertion occupying a slot. Unique insertion first checks whether the hash is already present. The distinction matters because approximate membership can influence whether a supposedly new item is accepted by a uniqueness-oriented operation. Exact application identity belongs elsewhere if rejecting a new item incorrectly would be unacceptable.

Deletion removes one occurrence. That is useful for applications whose membership population changes, but the filter still works with fingerprints. Deletion should target items the application knows it inserted, with explicit tests for the duplicate lifecycle. A successful membership probe should not become the application's only record of ownership or insertion history.

The command integration adds CF.RESERVE and a dedicated ACL category. Its reservation parser accepts bucket size, maximum relocation iterations and expansion. It rejects a zero capacity, with further bounds on the optional configuration values. This gives an operator specific controls over allocation and growth rather than requiring accidental defaults from first use.

CF.RESERVE seen-items 1000 BUCKETSIZE 4 MAXITERATIONS 10 EXPANSION 2

This reservation form appears in the integration tests. Its values need a workload evaluation and a documented decision about whether growth is allowed. The existence of expansion is especially relevant when a filter is chosen to keep memory predictable. A starting capacity is not necessarily the final allocation policy.

Persistence and compaction preserve different properties

The release adds RDB serialization, including DUMP and RESTORE, for Cuckoo filters. The versioned interface records settings, bucket and item counts, deletion counts and subfilter data in its serialized view. Restoring the object therefore reconstructs its internal state, rather than rebuilding membership from the original application items.

Growth adds subfilters to avoid rehashing the entire existing filter. The source describes their bucket counts in terms of the base count and expansion. Compaction tries to move occupied slots from newer subfilters into older ones, freeing a newest subfilter after emptying it. Its manual continuation mode keeps trying older subfilters when an earlier relocation attempt cannot empty one.

Memory evaluation should include a grow-and-delete cycle, since deleting items and releasing an allocation are separate events. The release notes describe automatic compaction after a deletion threshold and a manual CF.COMPACT command. That supplies an operation to test, without implying that every deletion immediately returns a corresponding number of bytes to the operating system.

For a filter-backed application, persistence testing should include duplicate items and the filter's configuration. A restored membership result alone would not verify that occurrence counts or future growth behave as expected. The internal state listed by the serializer makes those additional checks concrete.

WAIT observes progress across engine shards

The WAIT implementation accepts a replica count and timeout in milliseconds. The 1.40 server source rejects the command on a replica and returns zero immediately if the primary has no tracked replicas. It takes a snapshot of the replica objects before beginning its checks.

The primary then captures the current journal sequence number on each engine shard and emits a ping to prompt acknowledgements. A replica counts as caught up only when its acknowledged progress meets the captured target on every relevant shard. The checks run on the shards' own execution threads to read their acknowledgement state safely.

A source comment explains why this uses per-shard progress at call time instead of tracking the calling connection's last write. Journal writes occur after commit on shard fibers, decoupled from the issuing connection. Per-connection tracking would add bookkeeping to the write path. The implementation therefore waits for a broader set of already recorded writes than the calling connection's own operations.

WAIT 1 5000

This asks for one replica with a five-second timeout. The returned integer is the number that acknowledged the captured targets, and the caller must compare it with the requested threshold. The code does not turn an insufficient acknowledgement count into an automatic rollback of the preceding application write. I would make that outcome part of the caller's error and retry policy.

Compatibility includes bounded waiting

Inside MULTI/EXEC, the command returns the current acknowledgement count without blocking. Outside that context, the implementation polls while the count is insufficient and the deadline has not passed. Its interval is one hundred milliseconds. That matters when evaluating short requested timeouts and the granularity of observed completion.

A zero timeout uses a ten-minute maximum in this implementation. The source explains that this wait is outside the usual transaction blocking machinery and could otherwise retain a fiber indefinitely. The loop also exits for server-state changes or a role change, with additional guards when shard callbacks observe that the node is no longer primary.

The replica snapshot remains fixed for the call. If all still-running replicas in that snapshot have acknowledged and their count is below the requested number, the command can return early. A replica connecting later does not join the existing wait. Insufficient replica count, disconnection and takeover consequently need separate tests alongside the successful one-replica example.

The regression tests also exercise a request for zero replicas. That returns the actual acknowledged count instead of forcing the result to zero because the threshold is already satisfied. Another case requests more replicas than are tracked and checks that the operation returns after the available replica catches up. These cases make the integer result useful as an observation in its own right. A client wrapper should preserve the returned count so that application code can distinguish the achieved acknowledgement level from the level it originally requested.

These details define an acknowledgement operation. They do not supply a consensus protocol or a disk-flush confirmation in the shown path, which checks acknowledged journal positions. I would describe an application's guarantee using that actual observation and decide separately what recovery behaviour it requires. A successful count is useful evidence, but its meaning should remain tied to the implementation.

Vector storage follows declared width

Version 1.40 also stops always widening stored vectors to FLOAT32. The release lists native storage and distance computation for INT8, UINT8, FLOAT16, BFLOAT16, FLOAT32 and FLOAT64. A declared representation now influences the index's storage and calculation more directly. Memory measurements need the dtype planned for the application.

The new controls include EPSILON for HNSW range-query overscan and working EF_RUNTIME control. A fixed relevance set makes the latency and memory comparison interpretable when evaluating those controls. Reducing storage width and adjusting traversal controls can change different parts of retrieval, so combining both changes in one unexplained comparison would make the result harder to assess.

For my own selection process, the filter's collision tolerance and the caller's response to an insufficient WAIT result would be explicit design decisions. I would verify persistence after filter growth and exercise acknowledgement waiting during role changes. Vector adoption would get its own dtype-specific relevance and capacity measurements, with the declared width recorded alongside the results.