librdkafka 1.4.2 resets state when partition ownership ends
The maintenance release fixes hangs after topic recreation and stale offset commits after reassignment. Both repairs concern client state surviving a change that invalidates its meaning.
A partition can disappear and later return under the same topic name. A consumer can also lose a partition and receive it again after another consumer has made progress. librdkafka 1.4.2 repairs client behavior around both transitions. Its 6 May release announcement identifies a produce-and-consume hang after a partition disappears and returns, including topic deletion and recreation. It calls that hang a regression introduced in 1.3.0.
The other change resets stored offsets when consumer partitions are unassigned. That prevents a stored value from an earlier assignment being committed before the application consumes a new message. Both fixes concern retaining state beyond the lifetime in which that state was meaningful. They are client-library repairs, separate from an Apache Kafka broker release.
I value the control Kafka gives consumers over their positions in retained history. That control depends on a clear account of which position the client holds and when it is eligible to commit it. A partition name alone cannot establish that a local offset still represents current progress after ownership has changed.
Stored and committed offsets are different state
The released configuration reference describes the offset store as an in-memory value containing the next offset to commit for each partition. The automatic offset-store option controls updating that state as messages are delivered to the application. Automatic commit is a separate setting that periodically commits the stored progress.
Those settings allow the application to separate receiving a message from declaring its processing complete. The 1.4.2 introduction recommends disabling automatic offset storage for work that must finish before the offset becomes eligible for background commit. The application then stores the offset after processing, while automatic commit can remain enabled.
This illustrative configuration selects those two responsibilities separately. The application must still make the documented offset-store call after successful processing.
enable.auto.offset.store=false
enable.auto.commit=true
The distinction matters to the bug. A local stored offset is not the same thing as the coordinator's latest committed offset for the group. They can describe different moments, especially when another consumer owns the partition in between. Retaining the former through reassignment can incorrectly reintroduce an old progress value into the latter.
Reassignment exposed the stale value
The report in issue 2782 constructs a sequence with two consumers in one group and one partition. Initially, the first consumer stores progress without committing it to the group coordinator. A second consumer later receives the partition and commits further progress. When the first receives the partition again, a commit without explicit offsets can use the value it stored during its earlier assignment.
The issue discussion identifies the timing between assignment and fetching committed offsets as relevant. A commit made before the client has fetched the current committed position can still encounter the old local stored value. The maintainer notes that automatic commits can expose the same race, although the offset fetch usually completes quickly.
Resetting the stored value at unassignment gives the old assignment a definite end. A later assignment does not inherit an eligible local commit position merely because it reuses the same partition identity. The tagged changelog explicitly says the repair covers an offset-less manual commit and the automatic committer before a new message has been consumed.
I find this detail useful when interpreting what looks like an offset reset. The problem need not be the reset policy used when no valid starting offset exists. In this case, a stale local value can be committed again during an ownership transition. The resulting broker state is a consequence of the client's commit input.
Recreation requires the client to resume work
The topic-recreation repair addresses another invalidated state. The announcement says producing and consuming could hang after the partition went away and returned. It does not describe this as preserving the deleted topic's history. Recovery here means that the client can proceed when the partition is available again, not that deleting and recreating a topic retains its old records or consumer meaning.
That scope is important when applying the patch description. Restoring client progress through a metadata transition and recovering application history are different operations. The release repairs the former. The application still needs a coherent interpretation of the topic lifecycle and its offsets.
The same maintenance release handles previously unhandled per-partition OffsetCommit errors. That is another reason to distinguish requesting a commit from its successful completion for every partition involved. librdkafka 1.4.2 makes these transition and error paths more precise, so a current assignment can establish its own progress without committing a stored position left over from an earlier one.