librdkafka 2.16 rebuilds broker discovery during rebootstrap
Rebootstrap now discards stale broker connections and resolves bootstrap addresses again. The release also corrects lost-assignment commits and repeated outage reporting.
A connected broker is not necessarily a useful source of current cluster metadata. The librdkafka 2.16.0 release, published on October 6, fixes recovery paths that kept asking learned brokers for metadata even after those brokers reported that the client's metadata was stale. Another path reused an already connected bootstrap broker without resolving its address again.
The failure undermined the purpose of rebootstrap. A client can need to rediscover the cluster while old connections still exist. Restarting discovery must therefore change which brokers the client trusts for that process, including whether an old address remains valid.
Kafka's independent consumption and replay are useful only if clients can find the brokers serving the retained history. I see this as a substantive recovery correction. A client that remains alive but cannot replace its obsolete cluster view is operationally unavailable even when its process and some connections look healthy.
Bootstrap configuration is a discovery input
The configuration reference for 2.16.0 describes bootstrap.servers as the initial broker list. The client also supports brokers added through rd_kafka_brokers_add(). Rebootstrap repeats discovery using those inputs when its learned broker information is no longer sufficient.
The default metadata.recovery.strategy is rebootstrap. Recovery can start when all last-known brokers appear unavailable, when metadata cannot be refreshed within the configured interval, or when a metadata response requests it. These conditions include more than every TCP connection disappearing.
That distinction is important for an infrequently used client. The reference explicitly identifies a case where the broker set changes between metadata refreshes. The bootstrap configuration must remain a way to locate the current cluster even if the client's previously learned set has become obsolete.
The release's configuration names can be made explicit in an application configuration:
metadata.recovery.strategy=rebootstrap
metadata.recovery.rebootstrap.trigger.ms=300000
The interval shown is the documented default of five minutes. It is a trigger for inability to obtain metadata, not a promise that every outage resolves in that time. Successful rediscovery still requires reaching an appropriate bootstrap endpoint.
Recovery now replaces stale broker objects
The release notes say learned brokers and bootstrap brokers are decommissioned when a rebootstrap sequence begins. Bootstrap brokers are then recreated and connected again, resolving their addresses again. Only those brokers participate until a Metadata response rebuilds the broker list.
This changes the recovery state machine at the point where old information must stop controlling progress. Keeping a live connection to a broker from the stale view could otherwise make the client continue asking the same unhelpful source. Recreating the bootstrap connection also gives address resolution another opportunity to find the intended endpoint.
Coordinator connections receive related handling. When a transaction coordinator's broker is decommissioned, the coordinator resets so the connection and address resolution can be re-established. The notes say this aligns transaction-coordinator handling with the existing group-coordinator behavior.
Queued messages return to their partitions and can be sent once the new metadata identifies leaders. That step keeps message routing associated with the reconstructed cluster view. Rebootstrap is therefore not merely a request for another metadata response on whichever connection happens to remain open.
Administrative requests get a retryable transport result
An administrative request can be in flight when its broker is decommissioned. The release changes its error from RD_KAFKA_RESP_ERR__DESTROY_BROKER to RD_KAFKA_RESP_ERR__TRANSPORT. The notes identify the intended caller behavior as retrying the request instead of treating broker destruction as a hard failure.
The error classification matters because recovery changes internal broker objects intentionally. An application should not have to interpret that internal replacement as proof that its requested administration operation is permanently invalid. The transport result describes the interruption at a level where retry policy can act.
This does not establish that every administrative operation succeeded before the connection disappeared. It establishes which failure the caller receives during the described interruption. Operation-specific results still need to determine what happened when the request reaches the recovered cluster.
Outage reporting follows a bounded cadence
The same release reduces repeated ALL_BROKERS_DOWN events during sustained outages. Reporting is limited by reconnect.backoff.max.ms, and a recovered connection re-arms reporting for a new outage. Multiple rebootstrap cycles no longer imply an unrestricted stream of identical outage events.
That makes the callback's frequency different from the duration of the outage. An application must not interpret the absence of another immediate callback as recovery. A broker connection or successful operation supplies different evidence from a rate-limited error notification.
The configuration reference uses the same maximum backoff setting to bound reconnect delays. The release connects that setting to notification cadence as well, so changes to it can affect what operators see during a prolonged recovery.
Lost assignments no longer issue revoke-time commits
A separate consumer fix addresses automatic commits after assignment loss. On a session timeout or a maximum-poll-interval failure, the consumer marks its assignment lost and revokes its partitions. Previously, unassignment cleared the lost flag before the removed partitions were served.
That ordering defeated the guard intended to suppress commits for lost assignments. A commit could leave with an empty member ID and the previous generation. The release notes say the broker rejects it with UNKNOWN_MEMBER_ID, or with fatal FENCED_INSTANCE_ID for a static member.
The corrected code keeps the lost flag until removal completes, then clears it before the next assignment arrives. That preserves the distinction between losing authority over old partitions and completing the cleanup of those partitions. It also avoids leaving the flag set for a member that still retains other work.
Both fixes address stale client state during a transition. Rebootstrap replaces broker discovery state, while unassignment preserves lost ownership until cleanup finishes. Their practical value is that recovery can proceed using the current cluster and assignment information instead of an obsolete connection or prematurely cleared flag.