Kafka 2.6.1 keeps consumer heartbeats active during rebalancing

A consumer can still be alive while its group is negotiating a new partition assignment. Treating that interval as if the member had disappeared can create more coordination work before the existing rebalance finishes. Kafka 2.6.1, released January 7, includes a fix that allows consumer heartbeats during rebalancing. It is one of 41 issues addressed since 2.6.0.

The release notes list KAFKA-10122 for heartbeat behavior and KAFKA-10134 for high CPU use during consumer rebalance. These entries concern the client's coordination machinery. They are worth understanding separately from changes to a broker's handling of the records stored in the log.

I find Kafka's separation of retained data from consumer position useful. Several consumers can make different progress through the same history. The coordination around partition ownership needs the same precision, because an ownership change and a failed process are different events.

Membership has intermediate states

The consumer coordinator at the 2.6.1 tag distinguishes stable membership from preparing and completing a rebalance. Its rebalanceInProgress method checks the latter two states. A member consequently has meaningful coordination state even before it returns to a stable assignment.

The heartbeat thread checks whether the member has not joined the group or has encountered a fatal failure. In those conditions it disables heartbeating. Other intermediate membership states can continue through the loop. The distinction preserves a way for the coordinator to hear from a member that is participating in an ongoing rebalance.

This matters because the heartbeat is evidence about membership liveness. It does not itself complete the assignment protocol. A consumer waiting for that protocol can still maintain communication with the group coordinator. Conflating those operations makes a temporarily unfinished assignment look like an absent participant.

The code starts the heartbeat thread while ensuring that the group is active, after checking that the coordinator is ready. It then performs the needed join processing. This places heartbeat management alongside the coordination protocol, rather than making it a side effect of consuming a particular record.

A rebalance response is interpreted in context

The response handler treats REBALANCE_IN_PROGRESS according to the current state. If the consumer believed the group was stable, that response requests a rejoin and raises the error. If the consumer is already in a rebalance state, the handler logs that the response is being ignored and completes the heartbeat future.

Those branches express different information in the same protocol response. For a stable member, the coordinator is reporting a change the consumer needs to act on. For a member already taking part in rebalancing, the response is consistent with what it is doing. Repeating the same transition unnecessarily would not help complete the assignment.

The heartbeat loop also documents continuing after a rebalance-in-progress exception. It records receipt of a heartbeat so the member can remain in the group for the rebalance timeout. Its comment explains the failure it avoids, where stopping heartbeats lets the session timeout expire before the consumer can rejoin.

The distinction is therefore between time allowed for the coordination operation and the shorter liveness evidence needed while it happens. Maintaining the latter gives the former a chance to finish. The client is not claiming to have completed the rebalance merely because it can still communicate.

Application polling remains a separate obligation

The same loop checks the session timeout and the poll timeout independently. When the session timeout expires without a successful heartbeat, it marks the coordinator unknown. When the poll timeout expires, it treats the foreground consumer thread as stalled between calls to poll and may leave the group.

The diagnostic message names max.poll.interval.ms as the allowed interval and suggests either increasing it or reducing the batch returned through max.poll.records. This is a different condition from an ongoing rebalance. A background thread being able to send heartbeats does not establish that application code continues to make progress through its polling loop.

There is also explicit handling for a fenced instance identifier. The heartbeat thread records that as a failure, rather than treating it like an ordinary retryable heartbeat problem. These branches show why one generic rule about retrying coordination errors would discard useful distinctions in the protocol.

I regard that separation as the useful part of this maintenance release. The consumer can preserve membership during legitimate assignment work while still detecting an application that has stopped polling. Those checks answer different operational questions and need different state transitions.

The fix travels with the consumer

The relevant implementation is in the Java client coordinator. Updating a broker alone does not replace a client library packaged inside another application. A maintenance version can therefore matter to consumers independently of when an operator changes the broker deployment.

Kafka 2.6.1 contains other repairs, including transaction and Streams issues listed in its release notes. The heartbeat change has a specific effect within that wider update. It allows a member to remain observably live while rebalancing and preserves separate handling for stable membership, poll progress and fatal coordination failures.