Kafka 2.3.1 refines Connect's handling of work that keeps running
Kafka 2.3.1 continues the work behind cooperative Connect rebalancing. The worker distinguishes a received assignment from tasks already running and tasks stopped for reconfiguration.
An assignment received after a rebalance is not necessarily a list of tasks that need starting. Some tasks may already be running, while others may have stopped because their configuration changed. Kafka 2.3.1's maintenance work includes avoiding unnecessary connector and task stops. The distinction is central to making cooperative rebalancing preserve useful work in an actual worker implementation.
Apache lists the release as October 24. Its release notes identify KAFKA-8676 as avoiding unnecessary stops, alongside other Connect fixes. The tagged implementation shows the bookkeeping needed to distinguish retained work from work that requires a restart. A protocol's permission to keep tasks running is useful only if the worker's lifecycle handling preserves it.
I value the ability to build several independent systems around a Kafka stream. Shared integration infrastructure should not interrupt all those systems just because one connector changes. This maintenance work concerns the concrete execution state needed to keep that independence where assignments permit it.
Received assignments and running assignments differ
The DistributedHerder implementation in 2.3.1 keeps both the assignment received from the group and a running assignment. Its startWork method starts connectors in the difference between those collections. It uses the same kind of difference for tasks.
The helper constructs a set from the updated assignment and removes resources already represented as running. If the running collection is empty, the whole update is returned. That is the relevant execution decision, an assigned resource needs a start operation only if it is not already part of the running work.
This distinction follows from the cooperative model introduced in Kafka 2.3. KIP-415 permits workers to retain resources while joining the group and requires explicit revocation for work that must move. An assignment round can therefore include resources that survived the round intact. Starting every listed resource again would ignore that retained execution state.
The implementation also distinguishes the older protocol. After startup processing, it retains an empty running assignment for protocol version zero, while the newer protocol retains the current assignment. That difference reflects the eager protocol's assumption that work has been released during coordination and the cooperative protocol's ability to preserve it.
Reconfiguration creates a separate restart obligation
Task configuration changes need their own treatment because a task can remain assigned to a worker while requiring new execution state. The tagged code identifies local tasks belonging to connectors whose task configurations changed. It stops and awaits those tasks, then records them in tasksToRestart.
Before ordinary startup, startWork removes that recorded set from the running assignment. It then clears the restart set and computes the difference against the newly received task assignment. A stopped task is no longer incorrectly counted as running merely because it held that assignment before.
The code comment makes the second condition explicit. These tasks restart only if they appear in the assignment just received after rebalancing. A reconfiguration event does not authorize a worker to start a task that the group has since assigned elsewhere. Lifecycle bookkeeping must respect both local configuration work and current group ownership.
That is a useful example of why a single boolean such as assigned would be insufficient. A task can be assigned and running, assigned and stopped for reconfiguration, or no longer assigned. The start decision needs the current combination. Preserving work safely requires more information than simply suppressing all stop calls.
Configuration state must match the assignment
The same implementation checks the configuration-log offset against the received assignment. If the local configuration state does not match, it requests another group join rather than starting work against the wrong configuration snapshot. If behind, it attempts to read the configuration log to the required point.
That condition ties execution to the configuration used by the group. Avoiding unnecessary restarts would be a poor optimization if the tasks that remain or start used unrelated configuration. The worker needs a consistent basis for deciding what the assignment means before it declares the rebalance resolved.
My interest in these details is that they separate three observable facts without changing the stream's semantics. A resource belongs in the current assignment, an execution instance may already be running, and its configuration must match the expected state. The maintenance implementation uses those facts to decide which lifecycle operations are necessary.
Kafka 2.3.1 therefore does more than preserve the name of the cooperative protocol introduced earlier in the series. Its worker implementation retains running assignments, tracks tasks stopped for reconfiguration, and starts only the work still owned after the new assignment. Unaffected tasks can remain active while the worker accounts for the resources that actually changed.