Dragonfly stops expiration from inflating RSS eviction debt

A cache can delete the right keys and still make the wrong memory decision afterwards. Dragonfly's cache-mode regression did exactly that. Background expiration contributed to a counter intended to account for memory already deleted while waiting for RSS measurements to catch up. That counter kept growing even when there was no RSS eviction in progress. Eventually, subtracting it from the measured resident memory could underflow and produce an excessive eviction target.

Dragonfly 1.34.2, released on October 14, fixes the regression. Its release notes strongly advise upgrading deployments using 1.34.0 or 1.34.1 in cache mode. This is a narrowly scoped patch release with one listed change, but that change affects the policy deciding whether live cache entries should remain in memory.

I prefer Dragonfly when a larger machine can keep a cache simpler than a cluster. That preference makes memory management part of the architectural argument, rather than an implementation detail below it. A cache needs to discard entries under real pressure. Unrelated expiration history should not manufacture that pressure.

Two reasons for deleting keys

The issue describing the failure, opened on October 7, identifies the shared background path as the source of the problem. Expiration and eviction both run through the heartbeat logic. Expiration removes keys whose lifetimes have ended. Eviction removes entries to meet a memory target. Both operations free bytes, but those bytes do not always have the same meaning to the eviction controller.

In the 1.34.2 implementation, RetireExpiredAndEvict first calculates an eviction goal. As it visits databases, it performs expiration work and subtracts the resulting deleted bytes from that goal, without reducing the goal below zero. If a goal remains, the function performs eviction work and accounts for those deletions too. Expiring keys can therefore satisfy some of the immediate need to free memory.

That arrangement is reasonable. Memory recovered from an expired entry does not need to be recovered again by evicting a live one. The problem was in the lifetime of a second counter, deleted_bytes_before_rss_update. Before this patch, every heartbeat added its deleted bytes to that state, regardless of whether RSS-driven eviction was active.

The counter exists because the controller uses a sampled measure of resident memory as well as used-memory accounting. The implementation subtracts tracked deleted bytes from the RSS observation before calculating further eviction work. When a later observation shows RSS falling, it reduces the outstanding tracked amount, distributing the observed decrease across shards. This lets the controller recognize deletion work that has already occurred before the RSS measurement fully reflects it.

That is an estimate of pending effect. It is not a running total of every byte that expiration has ever removed. Accumulating ordinary expiration outside an RSS eviction interval gave it the latter meaning, while the calculation continued treating it as the former.

How a subtraction becomes eviction pressure

The issue points to an unsigned arithmetic failure. When the accumulated deletion estimate exceeds current RSS, subtracting the estimate does not yield a useful negative memory figure. It wraps to a large unsigned value. The eviction calculation then sees a value above the configured memory limit.

The helper calculating per-shard eviction work has a branch for usage above that limit. It requests enough deletion to cover the excess, plus the shard's memory budget threshold. That branch is intended for real overuse. Feeding it a wrapped result from an unrelated counter makes the requested work unrelated to the process's actual memory pressure.

This explains why the issue is more serious than a misleading statistic. The counter participates in a control decision that deletes entries. A cache can tolerate intentional eviction as part of its capacity policy, but excessive eviction can erase useful contents and make newly populated data disappear. The issue's title describes all keys being deleted with cache mode enabled, while its body gives the accounting mechanism behind that behavior.

I read this as a defect in the scope of the estimate. The meaningful question is how much deletion the current RSS decision has already initiated and has yet to observe. Carrying expiration history into that estimate changes the question, even if each individual expiration was correct.

Tracking follows RSS eviction activity

The fix backported into 1.34.2 introduces a track_deleted_bytes state flag. It begins false. When calculation produces a nonzero RSS eviction goal, tracking turns on. Heartbeat deletion work is added to the outstanding counter only while tracking is enabled.

Stopping tracking has two conditions. There must be no current RSS eviction goal, and the outstanding tracked deletion count must have cleared. Retaining tracking until that count clears preserves the accounting across the interval in which previously deleted memory is still being reconciled with RSS observations. The repair does not simply reset all state whenever one heartbeat happens to require no further eviction.

The patch also adds an underflow diagnostic and bounds the adjusted RSS input by the configured memory limit. If the subtraction still wraps, taking the minimum with that limit prevents the wrapped number from entering the helper as an arbitrarily large usage figure. This safeguard does not imply that no eviction will be requested. At the limit, the helper can still request work to restore its configured free-memory margin.

That distinction matters when describing the patch. The principal repair is restricting when deletion debt accumulates. Bounding the arithmetic is an additional defense in the calculation, not a replacement for giving the counter the right lifetime.

The regression test checks retained contents

The same patch adds a regression test with a 256 MiB memory limit and cache mode enabled. It repeatedly populates keys and gives alternating groups short expiration times. After those expiration cycles, the test clears the database, populates a fresh set of keys, waits for heartbeat work, and checks that the expected key count remains.

This sequence exercises the transition from accumulated expiration activity to a new cache population. It does not merely assert that a counter has a particular internal value. Its observable requirement is that the new entries survive when the background controller runs. The test is attached to the reported issue in the source, making the intended behavior of the fix concrete.

For an affected deployment, the release's upgrade recommendation is unusually direct. I regard that recommendation as justified by the deletion mechanism and the small scope of the patch. The regression test specifically requires the fresh population to survive after the earlier expiration cycles and a database clear. That catches the stale accounting across populations without relying on a particular internal counter value.