Redis 4.0 RC1 separates key deletion from memory reclamation

Removing a key from a database and releasing all the allocations behind its value need not happen at the same time. Redis 4.0's first release candidate makes that distinction available through UNLINK and asynchronous database flushing. The server can remove the visible data before a background thread finishes reclaiming its memory.

The 4.0 release notes date RC1 to 2 December 2016. The notes shipped with RC1 identify its internal version number as 3.9.101 and explicitly say it is not production quality. Its main feature set is frozen, with exceptions for changes between fixes and improvements. This is a candidate for exercising substantial core changes, not a declaration that the new version is ready for production.

I like Redis because its commands make operations on stored data explicit. An operation's cost matters as much as its syntax when the server is shared by many clients. Separating logical deletion from reclamation gives applications a more precise choice about when a potentially large amount of freeing work happens.

The amount of work depends on representation

The RC1 lazy-free implementation estimates how much work releasing a value requires. For strings, the estimate is always one. A hash-table-backed hash or set uses its entry count. A skip-list-backed sorted set uses its length, while a list uses the number of quicklist nodes.

Those details prevent a common misreading of asynchronous deletion. The implementation does not simply ask how many bytes the value occupies or how many logical elements a user thinks it contains. It estimates work from the physical representation. An object stored in one allocation can have a low estimated freeing cost even if its logical contents are substantial.

The source defines LAZYFREE_THRESHOLD as 64. In dbAsyncDelete, a value whose estimated effort exceeds that threshold is handed to a background job. Smaller values are freed synchronously because, as the comment explains, queuing a small amount of work would be slower. The asynchronous interface therefore includes an implementation choice about which values benefit from deferral.

This threshold is an implementation detail of RC1, rather than an application promise about object size. Its usefulness is explanatory. The new command avoids putting expensive value reclamation on the immediate command path, but it does not move every instruction associated with deleting a key to another thread.

The key disappears before expensive freeing finishes

The deletion function first removes any expiration entry and unlinks the key from the main dictionary. It then examines the value and, where appropriate, queues a BIO_LAZY_FREE job. Before releasing the unlinked dictionary entry, it clears that entry's value pointer so the value can survive until the background job frees it.

The server still performs the dictionary changes that make the key unavailable. Deferring reclamation does not mean leaving a readable key around until a worker reaches it. The ordering separates the externally visible removal from the later release of the detached value's allocations.

The worker eventually calls decrRefCount and updates the count of pending objects. The source exposes that pending-object count through lazyfreeGetPendingObjectsCount. This reflects an important resource consequence of deferral. A value can be absent from the keyspace while its memory is still awaiting reclamation.

The release notes describe UNLINK as the nonblocking counterpart to DEL. It takes the key to remove, as in this command example.

UNLINK cached-report

For an application deleting a large aggregate, this selects the new deletion path. It does not establish a fixed delay before memory returns to the allocator. The source defines the work queue and freeing mechanism without promising such a delay.

Asynchronous flush replaces the dictionaries

The same distinction applies to clearing a database. RC1 adds the ASYNC option to FLUSHDB and FLUSHALL. The implementation of emptyDbAsync creates new empty dictionaries for the database and its expiration metadata, then schedules the old dictionaries for background release.

This makes the transition concrete. The active database starts using empty dictionaries while the worker owns the old ones. Its later freeing work can traverse the detached dataset without keeping those keys in the active database. The cluster slots-to-keys map has a corresponding asynchronous replacement path.

The selected database determines which keys FLUSHDB removes. The new command form requests asynchronous reclamation.

FLUSHDB ASYNC

This is still a command that clears data. The new option changes the reclamation path, not which database the command targets. The release's implementation establishes the visibility transition before the old data structures are reclaimed.

I find this useful because it names a tradeoff directly. Redis can shorten the immediate work of deletion by retaining detached allocations temporarily. That distinction is more informative than treating an in-memory operation as automatically cheap simply because it avoids disk access.

The candidate changes extension and replication too

Lazy freeing is one part of a release that also introduces loadable modules. The RC1 module introduction describes dynamic libraries loaded at startup or through MODULE LOAD, using the C API in redismodule.h. It identifies API version 1 and warns that the module documentation describes an alpha API whose details may change.

The release notes also introduce PSYNC2. Its stated aim is to permit partial resynchronization after events such as a failover or replica restart, where the previous protocol could require a full resynchronization. The notes retain compatibility with older upstream servers while changing how chained replicas receive the replication stream.

These changes explain why the maintainers chose a major version number. The candidate modifies extension points and recovery behavior as well as memory management. Its release notes call for further testing before a final release, with the frequency and seriousness of reported bugs informing that decision.

For lazy deletion specifically, the available implementation is already precise. Redis removes keys from active lookup structures, estimates the cost of freeing their values, and defers sufficiently expensive reclamation. Database flushing can replace entire dictionaries before freeing the old set. Applications gain control over that distinction through explicit command forms in the candidate.