librdkafka previews broker coordinated consumer assignment
librdkafka 2.4 adds early access to KIP-848. Assignment moves to the broker, while callback behavior and static membership require attention from native-client applications.
A consumer group does not need one of its application members to calculate the partition assignment. KIP-848 moves that responsibility to the broker's group coordinator and delivers assignments through heartbeats. The change removes a client-side coordination role while also changing the contract applications use to respond to assignment updates.
librdkafka 2.4.0, released May 7, provides early access to that protocol. The release limits it to evaluation and explicitly excludes production use. Its versioned introduction documents both activation and behavior that differs from the classic protocol.
Kafka's partition ownership model is central to how I judge a consumer design. Moving assignment calculation into the broker does not remove the application's responsibility for handling changed ownership. The useful part of this preview is the explicit description of that new relationship.
Both sides must select the protocol
The guide requires a KRaft cluster running Apache Kafka 3.7.0 or newer, with the new protocol enabled through group.coordinator.rebalance.protocols. A new native client by itself is insufficient. The coordinator must support and enable the protocol the client requests.
On the client, the selection is explicit:
group.protocol=consumer
The guide also introduces group.remote.assignor to choose the broker-side assignor. That replaces the assignment-strategy configuration used by the classic protocol. The location of the setting reflects the changed ownership of assignment calculation.
The preview includes subscriptions to named topics, rebalance callbacks and static group membership. Regular-expression subscriptions and the proposed AdminClient changes remain listed as future work. This is a defined subset of the proposal, so describing the entire KIP as implemented would overstate the release.
Rebalance callbacks become incremental
The documented protocol is fully incremental. Applications are directed toward rd_kafka_incremental_assign and rd_kafka_incremental_unassign in rebalance callbacks. When a remote assignor is configured, those incremental functions are already required in this release.
The guide describes temporary compatibility for existing callback code based on the selected assignment strategy, with removal planned for another release. That compatibility should not obscure the intended contract. The application receives incremental changes and needs to apply those changes without interpreting every callback as replacement of the entire assignment.
All assignors are sticky under the new protocol, including range assignment. That differs from the older range behavior named in the guide. Choosing a familiar assignor name therefore does not establish that every detail of its previous rebalance behavior remains unchanged.
The broker also controls the consumer session timeout and heartbeat interval through group.consumer.session.timeout.ms and group.consumer.heartbeat.interval.ms. The corresponding classic client settings do not serve the same role after selecting this protocol. Configuration ownership follows the coordinator's expanded responsibility.
The guide also changes how topic availability appears during subscription. A topic missing from the local cache no longer immediately produces UNKNOWN_TOPIC_OR_PART through the same preliminary check. Subscription remains meaningful because the topic could be created afterward, while the coordinator controls the resulting assignment.
Duplicate static identities fence the newcomer
Static membership has a consequential change. If two consumers present the same group.instance.id, the later arrival receives the fatal UNRELEASED_INSTANCE_ID error. The guide contrasts this with fencing the existing member under the previous behavior.
This prevents two processes from repeatedly displacing each other under one identity. It also means a replacement for a crashed instance may have to wait for the old session to time out. Properly closing a consumer during shutdown becomes relevant to how quickly a replacement can use its identity.
The identifier must consequently represent one active instance at a time. A deployment that briefly runs old and new processes concurrently under the same identity has a protocol-level consequence. The preview documents that consequence directly instead of assuming that a restart policy and static identity always compose without delay.
Fetching and committing can diverge during coordinator loss
The guide says session timeout is managed remotely. If a member cannot reach the coordinator, it can continue fetching while unable to commit. It receives fencing information when it can obtain the relevant heartbeat response.
That differs from the classic behavior described in the same section, where the client stops fetching after its local session timeout. Continued delivery from partition leaders does not establish that coordinator-dependent operations are also succeeding. The application needs to distinguish those observations when interpreting progress.
I find this distinction particularly useful for operational reasoning. A consumer receiving records is evidence about its fetch path. A successful commit is evidence about the group's recorded progress, and the preview makes clear that those paths can have different availability during a coordinator problem.
The release therefore offers more than a new assignment location. It exposes an incremental callback contract, broker-owned timing settings and revised static-identity behavior. These are the concrete differences native-client users can evaluate with the explicitly enabled early-access implementation.