Redis 5.0.4 bounds malformed HyperLogLog data

A compact data structure still needs to distrust its encoded lengths. Redis stores HyperLogLog registers in either a dense representation or a sparse representation that compresses runs of register values. A malformed sparse run can claim more registers than the destination contains. Redis 5.0.4 adds checks before the affected loops expand or merge those values.

The 18 March release notes classify the update as high urgency. They describe stability failures under special conditions and specifically recommend upgrading installations exposed to untrusted users. The release includes several HyperLogLog corrections, plus a potential overflow fix in redis-check-aof. Understanding those changes requires distinguishing invalid representation handling from the statistical approximation that HyperLogLog intentionally performs.

I have used Redis since its release, and I like the usefulness of its specialized data structures. Their compact representation is part of that usefulness. This release interests me because implementation safety must hold even when stored bytes do not describe a legitimate instance of the advertised structure.

Sparse registers need a destination bound

The versioned HyperLogLog implementation describes two storage encodings. Dense storage packs register values into fixed-width fields. Sparse storage uses opcodes for runs of zeroes and repeated nonzero values. A parser advances through the encoded bytes while maintaining a logical register position.

That creates two different boundaries. The parser has to remain within the encoded input, and the expanded position has to remain within the logical register array. An encoded opcode may fit inside the source string while describing a run that extends beyond the array. Checking only the source pointer cannot establish the destination bound.

The corruption fix adds checks to both sparse-to-dense conversion and merge handling. Before iterating over a value run, the code adds its length to the current register index and compares the result with HLL_REGISTERS. If the run exceeds that limit, conversion frees its temporary dense allocation and returns an error. Merge handling also returns an error before expanding the invalid run.

The distinction between the two functions matters. A value can encounter representation conversion during one operation and register aggregation during another. Fixing the parser in only one place would leave another consumer making the same assumption. The patch addresses both loops shown in the commit, without requiring a claim that every HyperLogLog code path has identical parsing logic.

This failure is separate from approximate counting. A valid HyperLogLog trades exact cardinality for a small statistical summary. That tradeoff does not permit an encoding to address registers outside its allocated storage. Approximation affects the estimate returned to an application, while malformed input handling concerns whether the server can process or reject the object safely.

Counting has its own representation assumption

Another HyperLogLog commit expands the register histogram in hllCount to 64 entries. Its comment explains the decision to allocate enough space for register values that cannot arise from the intended hash result. Rather than assume all input values reflect valid construction, the histogram allocation accommodates the representable values.

That is a different defensive technique from rejecting an oversized sparse run. A larger histogram keeps an unusual register value within allocated memory. The run-bound checks reject an impossible logical extent before writing it. Both changes remove reliance on properties that ordinary commands would normally preserve.

The release also contains parsing fuzz tests for dense and sparse encodings. Their presence fits the defect class. Data-structure operations normally exercise legitimate encodings, but malformed inputs explore assumptions at conversion and counting boundaries. The test changes recorded in this release specifically target those representations.

I consider this an important distinction when assessing a maintenance release. A service can behave correctly for ordinary inputs and still require an urgent update because a client can cause it to interpret unusual state. The release's emphasis on untrusted users gives the affected deployment context without implying that every normal HyperLogLog request crashes.

An AOF checker also parses hostile material

The AOF diagnostic patch replaces two unbounded formatting calls with bounded snprintf calls. Both changes are in the checker's error-reporting macro. One formats a message into a temporary buffer, and the other prefixes that message with the file offset in the final error buffer.

This is more specific than saying the release changes AOF durability. The patch concerns formatting diagnostics while checking a file. It does not change the server's fsync policy or the ordering of acknowledged writes. Its importance lies in making the recovery tool's own error handling respect its buffer sizes.

The checker at the release tag reads the AOF's command representation and tracks an offset up to which it can process the file. It also recognizes an RDB preamble before continuing into the AOF tail. Its job requires handling files that may already be invalid, so the diagnostic path is part of the expected workload.

The optional repair mode can truncate an invalid tail after confirmation. The nonrepair invocation reports an invalid file without taking that truncation step. Inspection and modification are therefore separate choices in the tool itself. A parser failure has to be communicated safely in either mode, including when the user only wants to learn where processing stopped. That makes accurate parsing and diagnostics materially different from cosmetic logging. A user is assessing a file and may authorize a change to it based on what the tool reports. The bounded formatting change improves the tool executing that process.

Other 5.0.4 fixes cover stream delivery counts, sorted-set edge behavior and replication handshake misuse. Those changes do not share one failure mechanism. For HyperLogLog and AOF checking, the common practical issue is narrower. Redis has to interpret encoded data outside the assumptions of ordinary valid command construction, and the release adds explicit memory bounds in the affected implementations.