librdkafka can abandon stale broker discovery and bootstrap again

A long-lived client can retain a broker list that no longer leads to useful cluster metadata. Retrying those addresses is different from asking the original discovery service for a new list. librdkafka 2.11.0 extends automatic rebootstrap so a client can make that second choice after a metadata timeout or an explicit server response.

The July 3 release implements KIP-1102. Its versioned configuration reference explains the conditions and the source addresses used when discovery restarts. The feature builds on the existing recovery strategy rather than inventing a new application restart requirement.

I value Kafka's ability to keep consumers independent. A consumer that has been quiet for a long interval still needs a way to find the current cluster when it resumes. This is a recovery problem about discovery state, separate from the position it retains in a partition log.

The bootstrap list is a recovery input

The configuration reference describes bootstrap.servers as an initial list of broker addresses. The client uses it to discover the cluster. After discovery, metadata supplies the brokers used for ordinary operations.

Rebootstrap repeats that initial process using the configured bootstrap addresses and brokers added through rd_kafka_brokers_add(). The original discovery inputs therefore retain a role after startup. They are a route out of stale discovered state.

The documented use case is a client communicating so infrequently that the broker set can change completely before its next metadata refresh. In that case, retaining only the last discovered addresses may leave the client retrying a set that no longer represents the service.

The distinction is useful because increasing connection retries cannot change what addresses those retries target. Rebootstrap changes the source of discovery. It can obtain a new broker list if the bootstrap path still provides access to the current cluster.

That condition remains necessary. Rebootstrap does not create a reachable bootstrap service or correct an unrelated authorization failure. It retries the supplied discovery inputs, so those inputs still need to lead to the intended environment.

A metadata timeout supplies another trigger

The implementation patch adds metadata.recovery.rebootstrap.trigger.ms. The default is 300000 milliseconds. When the client cannot obtain metadata for that interval and its strategy is rebootstrap, it repeats discovery.

The following configuration spells out the strategy and the default timeout. The hostnames are illustrative discovery endpoints.

bootstrap.servers=bootstrap-a.example.test:9092,bootstrap-b.example.test:9092
metadata.recovery.strategy=rebootstrap
metadata.recovery.rebootstrap.trigger.ms=300000

The timeout concerns obtaining metadata. It should not be treated as an application message delivery deadline or a consumer session timeout. Those operations have different state and different failure meanings.

The reference lists three recovery conditions. All last-known brokers can appear unavailable, metadata can remain unobtainable for the configured interval, or a metadata response can request rebootstrap. The added timeout and server signal widen the situations in which the client can return to discovery.

This matters when the client has not concluded that every known broker is unavailable but still cannot get useful metadata. Recovery no longer depends exclusively on that one interpretation of the connection state. The client has an independent progress condition.

The recovery strategy remains explicit

The configuration accepts none and rebootstrap, with rebootstrap as the documented default. Selecting none means the client does not repeat bootstrap. The timeout's effect therefore belongs to the selected recovery strategy.

That relationship is worth retaining in documentation. A timeout value by itself does not override a strategy that disables rebootstrap. The strategy says whether the action is permitted, while the interval says when lack of metadata progress can trigger it.

My interest is in making those recovery inputs visible. Kafka's durable records do not imply that every client can discover their current leaders indefinitely using an old address list. The bootstrap configuration is part of the application's continuing operational dependency.

OAuth and compatibility receive separate changes

The release also implements KIP-1139 for the OAuth JWT bearer grant. The versioned reference exposes the grant type alongside assertion configuration, including signing algorithm and private-key inputs. Its default grant remains client credentials.

That is a separate authentication capability. A client can discover the correct brokers and still need valid credentials to use them. Conversely, valid credentials do not help a client whose discovery state never reaches the current service.

The release also stops deriving balanced-consumer and GSSAPI feature support from JoinGroup version zero. The notes explain that this old API version is absent in Kafka 4.0. Feature detection needed to recognize the newer supported protocol arrangement instead of interpreting one removed version as absence of the broader capability.

A buffer-counter data race and a per-queue poll-ratio calculation also receive corrections. Those maintenance items concern the native client's concurrent state and telemetry, independently of enabling a new authentication grant.

For this release, rebootstrap supplies a concrete recovery action for stale discovery. The client can return to its bootstrap addresses when metadata progress times out or the server requests it. The retained partition position and the refreshed broker list then serve their separate purposes in resuming consumption.