librdkafka adds leader epochs to consumer position tracking

A consumer offset can fall within the current log and still refer to a position whose earlier history has changed. After truncation and subsequent appends, checking only whether the numeric offset is in range misses that distinction. librdkafka 2.1.0, released April 6, adds KIP-320 support so fetchers can detect and handle log truncation. The change gives the client more information about the history associated with its position.

This matters when an application treats consumed records as durable history. A position is useful only in relation to the partition history it describes. Detecting a divergence lets the client respond to that changed history instead of silently continuing because the next requested number happens to exist.

I find replay one of Kafka's most useful properties. That makes the interpretation of a saved position important. The offset says where consumption has reached, while leader-epoch information helps establish whether the current leader's log still supports that position's history.

A valid number can conceal divergence

Jason Gustafson's KIP-320 proposal describes the problem after an unclean leader election. Committed data can be lost. A consumer near the old end may encounter an out-of-range offset, but new records can also extend the replacement log far enough to make its requested offset valid again.

The proposal uses leader epochs to distinguish those histories. After observing a leader change, a consumer with the relevant epoch information can query OffsetForLeaderEpoch before continuing. This identifies divergence that an offset-range check alone would miss. The design also extends committed-position information so epoch context can survive a restart or rebalance.

The point is detection of changed history, not recovery of records that no longer exist in the cluster. A client can choose a valid continuation and report the condition. It cannot reconstruct lost application events solely from an offset and epoch. That distinction keeps the feature's benefit precise.

The client exposes the additional information

The tagged 2.1.0 changelog lists new APIs for getting and setting the leader epoch on topic-partition entries. It also adds an API for retrieving a message's leader epoch. Assignment and partition-seek operations can accept partitions carrying an epoch, and time-based offset lookup returns epoch information.

Those APIs matter for applications that manage positions explicitly. Persisting only a number discards the extra history context the client can supply. An external position store needs a representation that can carry the information used when the application resumes. The feature is therefore relevant beyond the client's internal fetch loop.

The release also adds per-partition statistics for the current, stored and committed leader epoch. These correspond to different stages in the position lifecycle. Exposing them makes it possible to inspect whether the position information being retained has the epoch context expected by the consumer.

I prefer this explicit representation to treating an offset as a complete identity for consumed history. It gives application authors a more accurate description of what their stored progress means. The additional value belongs with the partition position, rather than in unrelated operational notes that the resume path cannot use.

Reset policy remains an application decision

The versioned configuration reference documents auto.offset.reset. Its choices include moving to the smallest or largest available offset and reporting an error. Those choices address what to do when there is no initial stored position or the desired offset is out of range.

The configuration reference uses error for the error-reporting option. That is worth naming because a description borrowed from another client's configuration can introduce the wrong value. A setting must match the library that parses it, even when the underlying Kafka concepts are shared.

Detection and policy perform different jobs. The client establishes that its desired continuation needs attention. The configured policy determines whether it adjusts automatically or reports the condition for application handling. Neither choice by itself explains how a downstream system should reconcile effects already produced from earlier records.

Position handling receives related repairs

The release fixes batch-consumption behavior around pause and resume, including incorrect stored positions and skipped offsets. It also corrects the timeout passed through partition-seek handling and a reference-count problem that could block consumer closure. The latter occurred when a partition was lost through forced unassignment or deletion of its topic. These changes concern the correctness of transitions between consumption states.

The tagged changelog still lists a batch-API interceptor issue during seek, pause, resume or rebalance operations. That is the release's own stated limitation. The new epoch APIs and truncation support improve the description of consumption progress, while the surrounding fixes address how the client maintains that progress through ordinary control operations.