Librdkafka exposes consumer rebalance history

A consumer that stops receiving records may be waiting for an assignment rather than waiting for new data. Those conditions require different explanations, but an application-level throughput graph can make them look identical. Librdkafka 0.9.3 adds statistics that expose part of the consumer-group lifecycle, including rebalance history and current assignment size.

The 23 January maintenance release lists rebalance_age, rebalance_cnt and assignment_size as new group statistics. It also fixes several coordinator waiting and assignment problems. The combination is useful because it improves both state-transition behavior and the evidence available when that behavior needs diagnosis.

I find Kafka's independent consumer positions useful, especially when a retained stream supplies several systems. Group membership adds another responsibility around that model. A consumer needs to know which partitions it currently owns before its position in those partitions can explain its progress.

Statistics describe the client's group state

The statistics emitter in the release source places the new fields inside a cgrp object when a consumer-group object exists. That condition matters. A producer or another client arrangement should not be assumed to emit meaningful group-assignment state merely because the statistics format contains other familiar fields.

The emitter calculates rebalance age from the current clock and the stored rebalance timestamp. When no timestamp exists, it emits zero. A zero value therefore needs context. It does not independently establish that a rebalance has just completed, because the implementation also uses zero when the timestamp is absent.

The count and assignment size come from the group's tracked state. Read together, they provide more information than a throughput measurement alone. An assignment size of zero helps explain why there may be no partition work to consume. A changing rebalance count indicates group lifecycle activity even if the application's own record processing has not changed.

These are observations of a particular client instance. They should not be treated as a complete broker-side account of every group member. A process restart also changes the context in which local statistics are interpreted. Monitoring needs to preserve the identity of the instance whose state it is comparing over time.

Statistics must be enabled and consumed

The configuration reference at v0.9.3 says statistics are disabled when statistics.interval.ms is zero. An application also needs to register a statistics callback. Installing the new client does not automatically deliver these fields into an external monitoring system.

The documented emission granularity is one second. A requested interval should therefore be understood within that implementation's timing limits, rather than as a promise that the callback supplies arbitrarily fine observations of each state change. The statistics summarize state periodically.

The emitter enqueues a statistics operation for the application after it builds the JSON. That is another distinction between generating information internally and having application code receive it. A program must integrate the client's callback processing into its lifecycle if it expects to use the new observations.

The result is a more specific diagnostic question. Instead of asking only whether records are arriving, the application can ask whether it has an assignment and whether group activity has recently changed. Those observations narrow an investigation without establishing the cause of every rebalance.

Coordinator fixes address stalled transitions

The release repairs a case in which recurring coordinator query errors could leave a consumer stuck waiting for a coordinator. It also adds timeout handling for operations in the coordinator-wait queue. A network or coordinator problem needs to remain a recoverable state transition, rather than indefinitely preventing queued work from progressing.

Other listed changes strengthen assignment and commit waiting. The notes also describe asynchronous termination by state to avoid recursive locking during timeout scanning. These are implementation-level coordination fixes, separate from the logic the application applies to each record it receives.

The heartbeat changes are similarly specific. The release sets a heartbeat request's timeout to the session timeout and allows at most one heartbeat request in transit. These rules relate liveness communication to the client's group session, rather than treating heartbeats as unbounded independent requests.

The configuration reference describes session timeout as the group's failure-detection timeout and heartbeat interval as its keepalive interval. Neither setting can be understood solely as a throughput knob. They influence how the group maintains membership while record consumption proceeds.

Producer shutdown receives a separate correction

Librdkafka 0.9.3 also fixes a failure to enforce the timeout in flush(). That is distinct from the consumer-group additions, but it reinforces the importance of bounded client lifecycle operations. A caller supplying a timeout needs the implementation to honor the bound when outstanding work does not finish.

The release includes a producer queue-size fix for platforms with a 32-bit size_t. That item concerns representation of the configured byte limit, not a change in the application's topic partitioning. Small client-version changes can therefore affect practical resource limits and termination behavior without changing the Kafka log model.

I prefer these explicit observations over inferring coordination state from processing rate. Rebalance count, age and assignment size do not replace application metrics, but they explain another part of the consumer's ability to work. In this release they accompany fixes to the waiting states that can otherwise leave a consumer alive while unable to obtain or use its assignment.