librdkafka moves the new consumer protocol into preview

Moving consumer coordination to the broker changes which settings a client can meaningfully control. librdkafka 2.10.0 makes that visible by rejecting several familiar properties when the new consumer protocol is selected. Its preview is therefore an implementation transition with configuration consequences, not simply a faster version of the existing consumer.

The April 17 release advances KIP-848 from early access to preview. It adds regex subscriptions, client-generated member identifiers, and support for inspecting the new groups. The release explicitly says the implementation is feature complete but the contract may still change before GA, and should not be used in production.

I regard partition assignment as one of the main decisions in a Kafka system. A client protocol that changes how assignments progress deserves an explicit adoption choice. The release's preview status is specific to this native client implementation, even though Kafka's Java-side consumer protocol has already reached GA.

Configuration now reflects who owns coordination

The KIP-848 design moves coordination away from thick clients and toward the group coordinator. Its incremental assignment model avoids requiring every member to pass through a group-wide synchronization barrier for every change.

That design explains the configuration changes in librdkafka. With group.protocol=consumer, setting partition.assignment.strategy is an error. The property belongs to the old client-side assignment arrangement and no longer selects the mechanism used by this protocol.

The release also disallows session.timeout.ms, heartbeat.interval.ms, and group.protocol.type for the consumer protocol. Copying an existing client configuration and changing only the protocol selector can therefore fail because the copied settings no longer belong to the selected contract.

The protocol opt-in itself remains concise. A preview configuration must omit the incompatible properties rather than retain them as harmless documentation.

group.protocol=consumer

That is an important difference between an ignored option and a rejected one. Rejection tells the application that its configuration expresses an unsupported assumption. Silently retaining the value could leave the operator believing the client still controls a timeout or assignor that the new protocol handles differently.

The release does not make these properties invalid for every classic consumer. Its statement is conditional on selecting the consumer protocol. Preserving that scope avoids turning a migration rule into an unrelated configuration ban.

Administration needs the target assignment

The release extends rd_kafka_DescribeConsumerGroups() to understand consumer-protocol groups. Its new group-type field distinguishes classic and consumer groups. A target-assignment field applies to consumer groups and otherwise defaults to null.

That field fits the protocol's reconciliation model. A desired assignment and the members' current state can differ while a transition is in progress. An administration interface that exposes only the older group shape would omit information needed to explain that interval.

Group configuration is also supported through the configuration administration APIs named by the release. The client can describe and alter group configuration through those operations. Administration is therefore part of protocol support, alongside the ability to fetch records.

Regex-based subscriptions and topic authorization errors in heartbeat responses complete more of the application-facing behavior. A client needs to express subscriptions and report failures in the protocol carrying its group state. Those are necessary parts of moving from an early implementation toward a usable contract.

Metadata handling repairs benefit ordinary clients too

The release includes maintenance changes independent of opting into the preview. It stops removing topics from cache on temporary metadata errors and reserves permanent treatment for topic authorization errors in metadata responses. A transient control-plane problem should not automatically become a permanent statement that an application's topic is gone.

Another fix addresses metadata refreshes enqueued on an unreachable broker. Retrying those requests could keep the full-metadata request counter from returning to zero, preventing discovery of a new controller or coordinator. The correction avoids retrying those particular requests in the way that blocked progress.

That mechanism explains why recovery could stall even when the client needed information about a different reachable component. A request directed to the unavailable broker could prevent the refresh needed to discover the replacement. The fix changes the dependency between those operations.

Consumer leader changes also lose unnecessary delays. The notes describe resetting fetch backoff on a new broker and removing a separate wait after leader change and offset validation. These are specific corrections to timing state retained across transitions.

My interest in those fixes comes from Kafka's replay model. An independent consumer needs to recover its connection and resume its position without mistaking a temporary metadata problem for permanent absence. The log's durability and the client's ability to discover its current leader remain separate responsibilities.

Preview is the native client's readiness statement

The release is feature-complete for the previewed protocol, but it explicitly reserves the possibility of small contract changes. That qualification is useful for framework and binding authors because their exposed configuration and administration types may depend on the native library's behavior.

The release also rejects ambiguity in settings instead of carrying old assumptions into the new protocol. A consumer using the preview must select the protocol deliberately and supply a configuration consistent with its ownership model. Classic consumers receive the metadata and recovery fixes without that selection.

For this version, the new capability is a more complete native implementation available for evaluation. Regex subscriptions and group inspection make the protocol easier to exercise, while the explicit configuration errors identify where the older client contract no longer applies.