Kafka 3.2.1 keeps transaction markers out of ordinary key compaction
A collision between application keys and transaction-marker keys could remove control records too early. The maintenance release repairs the cleaning decision and several other state-transition failures.
A transaction marker is a record in the log, but its key does not have the same meaning as an application's key. Kafka 3.2.1 fixes a cleaning error that failed to preserve that distinction. The July 29 release summary lists 13 fixes. The archived notes include KAFKA-14055, which concerns transaction markers lost when application keys conflict with marker keys.
The problem is consequential because compaction must preserve the information needed to interpret retained records. Keeping application data while removing the control record that establishes its transaction state can change what that data means. A log retaining bytes is insufficient if the associated control information disappears prematurely.
I value Kafka as durable history that consumers can interpret independently. This defect concerns the integrity of that interpretation. Its correction matters to the storage contract even though the release introduces no new producer or consumer interface for it.
Marker keys collided with application keys
The original issue describes hanging transactions observed in Streams changelog topics. The investigated keys had the same four-byte shape as keys in transaction control records. During offset-map construction, Kafka correctly excluded marker keys. During cleaning, it failed to exclude them from the ordinary retention decision.
That mismatch allowed a later application record with a matching key to make the marker appear obsolete. The cleaner could remove the marker before removing the transaction's corresponding data. The issue identifies hanging transactions as a symptom and loss of transaction atomicity as the more serious consequence.
The failure therefore needs both kinds of record and a cleaning interaction. It should not be described as every transaction losing its marker or every topic corrupting its data. The useful trigger is an application key conflicting with a marker key at a higher offset during compaction.
The cleaner distinguishes control batches
The 3.2.1 LogCleaner implementation makes the distinction visible in shouldRetainRecord. After checking whether the batch's records should be discarded, it retains records in a control batch. Ordinary records proceed to the key-based retention logic.
That key-based path looks up the latest known offset for a record's key. A record can remain when it is the latest relevant value or when deletion-retention rules require it. Transaction control records need their own decision before reaching that comparison because a later application value cannot substitute for a transaction marker.
The correction does not imply that all markers must remain forever. Transaction and batch-discard decisions remain in the surrounding cleaner. The particular safeguard prevents an ordinary application-key comparison from prematurely deleting a control record that the transaction-aware logic retained.
This is the important design distinction in the patch. Both record types share storage, but their keys belong to different interpretations. A generic optimization over key equality is safe only where that equality has the intended meaning. The record type supplies necessary context for that decision, even when both keys have the same bytes and occupy the same partition log.
Maintenance also addresses coordination boundaries
The release notes include a cooperative-rebalance fix involving offset commit during onJoinPrepare. They also list an AlterISR request failing to retry a retriable error. These descriptions concern transitions in ownership or coordination, where successful operation depends on more than the normal request path.
Another fix concerns validateOnly for partition creation under KRaft. The requested operation is supposed to validate a change without applying it. Correct handling therefore preserves a distinction between examining an administrative request and mutating cluster state. The release notes identify that distinction as broken in the affected path.
These fixes are independent of the marker-key collision. Their presence explains the value of a branch maintenance update for operators who are not seeking a new capability. Existing operations need to preserve their contracts during rebalances and administrative transitions as well as during steady execution.
My preference is to explain a patch through those concrete conditions. Saying a release improves stability hides the record types, callbacks and request modes that determine whether a particular workload exercises a defect.
The retained history needs its control information
Kafka 3.2.1 also repairs source-task offset commits and a memory leak when error tolerance is configured to allow all errors. It fixes OAuth token refresh with SASL extensions and a dynamic broker-certificate update regression. Those notes identify additional execution paths affected by this maintenance release.
The storage fix remains the clearest architectural example. Compaction cannot decide the fate of a transaction marker solely through the latest application record with equal key bytes. By separating control-batch retention from ordinary key retention, the cleaner preserves the transaction information needed to interpret the application records that remain.