Kafka 3.6.1 repairs Connect assignment and offset handling

A streaming process can stop making progress even when its records remain available. Kafka Connect needs a worker assignment for each task and a usable view of source offsets to resume work correctly. Kafka 3.6.1 repairs failures in both areas. The December 7 release entry lists 30 fixes since 3.6.0. Among them are a rebalance setting that could leave work unassigned and malformed offset metadata that could interfere with otherwise valid records.

These failures deserve more attention than their placement in a maintenance release suggests. A durable log preserves data, but a connector also needs to know who should process it and where processing should resume. The release notes include KAFKA-15693 and KAFKA-15800, which concern those separate parts of Connect's execution. Their remedies belong to the worker runtime, beyond the broker's responsibility to retain topic records.

I value Kafka's non-destructive consumption because retained data remains available to another reader or a recovering process. That makes progress metadata especially important to my view of the system. Keeping the bytes provides the opportunity to recover, while a task assignment and a correct position determine whether a connector can use that opportunity without unintended repetition.

A zero delay could suppress the next assignment

KAFKA-15693 concerns Connect's incremental rebalancing. A worker can lose an assignment during a rebalance before another worker receives that work. Connect supports a configurable delay before it resolves that imbalance. The issue identifies scheduled.rebalance.max.delay.ms as the relevant setting.

Setting this value to zero should remove the wait. In the defective behavior, lost assignments could instead remain unassigned after the rebalance. With scheduled delays disabled, the expected follow-up rebalance did not occur on its own. An unrelated event, such as a worker joining or a connector being created, could trigger a rebalance and allow the work to move again.

That mechanism explains why a process can appear to recover after an unrelated administrative change. The change supplies a missing trigger. It does not establish that the original connector configuration or destination caused the failure. The repair restores the intended meaning of zero delay so that assignment recovery does not depend on another event happening later.

This bug also distinguishes task recovery from task execution. A connector cannot retry its destination operation while no worker owns its task. Increasing a destination timeout or repairing destination credentials would address different parts of the path. Here the relevant evidence is an assignment that disappeared during rebalancing and did not return when the configured waiting period was zero.

One malformed key could affect the rest of a batch

KAFKA-15800 describes a failure while Connect reconstructs source-offset state. Deserializing a malformed offset key could raise a DataException inside the offset backing store's consumer callback. The exception propagated to a catch around processing a batch of records. As a result, records after the malformed entry in that batch could be omitted from the in-memory offset view.

The issue describes a consequence for tasks using that view. They might fail to see offsets that actually existed in the topic, allowing duplicate source records to be emitted. This is a failure of reconstructed runtime state. It should not be interpreted as evidence that the malformed key deleted the following records from the Kafka log.

The 3.6.1 OffsetUtils implementation contains the repair at the deserialization boundary. processPartitionKey catches a DataException, logs a warning and returns for that key. The method also checks the expected key structure before using it. A malformed key can consequently be ignored locally without aborting the surrounding batch processing.

The expected structure contains a connector name and a source-partition map. Those values identify which connector partition the offset metadata describes. The patched method checks that the deserialized object is a list of the expected length and that its elements have the appropriate types. These checks keep invalid input out of the map used to track connector partitions.

I prefer this repair's narrow scope because it preserves information from other valid records. An invalid entry should not prevent a worker from learning the positions of unrelated source partitions. Kafka's retention model makes rereading possible, but discarding useful offset evidence during reconstruction creates avoidable ambiguity about what needs to be read again.

The patch reaches beyond broker binaries

The release also includes a Java producer improvement, KAFKA-15415. A retried batch can skip its ordinary retry backoff when the client knows that the partition has a new leader. The issue explains that waiting through the old backoff is unnecessary once updated metadata points the retry at that new leader. This concerns producer latency around leadership changes, with the behavior implemented in the client.

Together, these changes show why the Kafka version of a deployment is not described fully by its brokers. The Connect runtime must contain the assignment and offset repairs. Java producer applications need the client implementation for the retry improvement. Updating broker binaries alone cannot replace code running inside an independent worker or producer process.

For Connect, the concrete benefit of 3.6.1 is a more reliable account of pending work. Zero rebalance delay no longer leaves lost assignments waiting for an unrelated trigger, and an invalid offset key no longer prevents processing the remaining valid entries in its batch. Both fixes make retained data usable by the processes responsible for moving it.