Kafka Connect stops revoking unrelated work during rebalances

Changing one connector should not require every unrelated connector in a shared cluster to stop. Kafka Connect's original assignment protocol imposed that disruption because workers released all their work before joining a rebalance. Kafka 2.3, released June 25, introduces incremental cooperative rebalancing. Workers can retain existing work while the group determines which particular assignments need to change.

KIP-415 explains why Connect makes the cost especially visible. A worker group hosts heterogeneous connectors and tasks, often belonging to unrelated integrations. A configuration change for one integration can trigger group coordination even when the others have no reason to move. The old protocol made group coordination a global execution interruption.

I value Kafka's independent consumers because several systems can use one stream without sharing a reading position. A shared Connect cluster should preserve a similar degree of independence between integrations where its assignment rules permit it. The new protocol reduces how much unrelated work has to stop during a change.

Joining a group no longer means surrendering all work

Before this change, a Connect worker stopped all its connectors and tasks in preparation for joining the group. The leader then computed an assignment from scratch. That established a simple synchronization barrier, but it made every rebalance disruptive to every worker's running resources.

Under KIP-415, the worker keeps running its assignments when joining. Its subscription metadata reports the work it already holds. The leader uses that information to compute both assignments and explicit revocations. A worker stops the revoked resources, removes them from its reported allocation, and rejoins so the next assignment can account for their release.

This changes the meaning of a rebalance without removing group coordination. The group may need successive rounds to move a resource safely. In one round the current owner is told to release it. In a subsequent round another worker receives it after that release is reflected in the reported allocation.

The proposal deliberately accepts that some scenarios require more rebalance rounds. The goal is to reduce the amount of execution interrupted by each round. Counting rebalances alone would therefore give an incomplete picture of the change. A round that leaves most tasks running has a different cost from a round that stops the whole cluster.

The assignment format adds revoked resources and a scheduled delay alongside the ordinary assignment. The subscription adds the current allocation, giving the leader the information needed to preserve work across rounds.

The leader still responds to the familiar triggers, including configuration changes and workers joining or leaving. The protocol does not make those events disappear. It changes the resource-revocation behavior used while responding to them.

Release must still precede reassignment

The old global barrier supplied a useful safety property. A task stopped and saved its progress on one worker before beginning on another. The cooperative protocol retains that ordering for a task that actually moves. It avoids imposing the same execution barrier on every unrelated task.

That is why explicit revocation and the second round matter. Assigning a task elsewhere while its previous owner still runs it would violate the ownership relationship the rebalance is supposed to establish. Incremental coordination needs evidence of release for the moved resource, even while other resources remain active.

Connect can make this change without exposing a new global barrier to connector implementations because tasks already run independently. KIP-415 says one task stopping and resuming does not require every task belonging to the same connector to do so. The proposed scheduling unit remains the individual connector or task assignment.

The implementation also benefits from existing separation between worker execution and coordination. DistributedHerder threads drive membership and rebalance processing independently of the threads running connectors and tasks. The control work can therefore determine changed assignments while unaffected execution continues. The new protocol uses that separation instead of requiring every task to participate in a new application-facing interface.

A short absence need not cause immediate redistribution

KIP-415 also introduces deferred resolution of imbalance. If a worker disappears briefly, the group can retain an imbalance for a bounded period instead of immediately distributing all missing work and then moving work again when the worker returns. This is useful for both unexpected process replacement and a planned rolling restart.

The control is scheduled.rebalance.max.delay.ms, whose default is 300000 milliseconds, or five minutes. It is the maximum delay the leader may communicate in an assignment, rather than a guarantee that every rebalance waits five minutes. The proposal permits an actual delay less than or equal to that maximum.

connect.protocol=compatible
scheduled.rebalance.max.delay.ms=300000

These are Connect worker settings. The first enables negotiation of the new protocol while retaining compatibility with the old one. The second sets the maximum deferral used to tolerate worker departures. They describe different decisions, protocol availability and the time allowed before redistributing absent work.

The waiting period has a cost. Tasks on the missing worker are unavailable while the group leaves their assignments unresolved. The intended benefit is avoiding unnecessary redistribution when that worker returns promptly. Tasks retained on healthy workers can continue, so the cluster accepts a localized temporary reduction in running work instead of a global interruption.

If the worker does not return within the delay, a later rebalance assigns the missing resources to surviving workers. The design therefore bounds its willingness to wait for the earlier arrangement. Deferred reassignment is a policy for recovering ownership, not an assertion that lost tasks continue to execute somewhere.

Leader changes limit what the group can infer

The proposal's examples distinguish an ordinary worker departure from a leader departure. Assignment heuristics use the leader's continuity across rounds. A new leader may not know whether an unaccounted resource is newly configured or was lost with another worker. That limits how confidently it can delay or restore a previous arrangement.

When the leader leaves and no delay is active, the new leader computes assignments without waiting. When a delay is already active, the new leader honors that delay. The proposal also explains that returning workers are not guaranteed to receive exactly their earlier assignment if leadership has changed during the interruption.

Those qualifications keep the feature's promise precise. It reduces unnecessary movement under the described conditions, but it does not establish permanent affinity between one worker and one task. The allocation still belongs to the group, and the group must converge after failures with the information available to its leader.

Compatibility belongs to the worker group

The new connect.protocol setting has eager and compatible modes in this proposal. Eager retains the original protocol. Compatible offers both and prefers the incremental cooperative protocol when the group supports it. A rolling worker upgrade can therefore retain service while the group transitions between protocol capabilities.

An eager worker joining the group forces a downgrade to the old behavior. The proposal describes a one-time double rebalance during that downgrade so workers release resources before returning to ordinary eager assignments. The presence of an upgraded binary on some workers consequently does not establish that the group is using cooperative execution.

This change is specifically Kafka Connect's protocol. It should not be described as enabling cooperative assignment for every ordinary Kafka consumer application in 2.3. Connect uses Kafka's group membership API, but it has its own resources and embedded assignment protocol. The scope is the worker group managing connectors and tasks.

Configuration updates become incremental in a separate API

Kafka 2.3 also adds incremental configuration updates through the administrative API. KIP-339 explains that the older replacement API required callers to know the full configuration. A read-modify-write cycle could lose another caller's update, while sensitive values could not even be read back to preserve them.

The new API accepts specific set, delete, append and subtract operations. It preserves all-or-nothing updates for a single configuration resource, without claiming a transaction across every resource in a request. This is separate from Connect assignment, but it addresses a related operational problem, changing one part of a system without replacing unrelated state.

My interest in Connect's new behavior is equally specific. The group can coordinate a change while healthy workers retain work that does not need to move. A resource that does move still has an explicit release-before-acquisition sequence, and absent work is redistributed after the permitted delay. The protocol reduces disruption while preserving the ownership condition that makes task reassignment safe.