Kafka 3.7.1 preserves MirrorMaker checkpoint knowledge across restarts

Restarting a replication task should not unnecessarily discard its ability to translate earlier consumer positions. MirrorMaker's checkpoint task had that problem because part of its knowledge lived only in memory. A restarted task could translate positions in newly mirrored records while losing the ability to translate positions it had previously understood.

Kafka 3.7.1, released June 28, includes a repair for that behavior. The release notes identify KAFKA-15905 among the maintenance fixes. Its account explains why recovery of checkpoint state matters independently of whether the replication process is running.

The durable-history model is what I value in Kafka. A second cluster is useful only if the systems consuming that history can identify meaningful positions there. MirrorMaker's offset translation is therefore part of replication's operational behavior, alongside copying the records themselves.

The task lost its previous checkpoint constraint

KAFKA-15905 describes the lost checkpointsPerConsumerGroup state when MirrorCheckpointTask restarts. That loss restricts translation to records mirrored after the latest restart. The issue's example contrasts an earlier translated position with a later one that remains possible after more records are mirrored.

The history behind the restriction matters. Earlier work prevented translation from an incompletely initialized offset-sync store, since incomplete knowledge could make a checkpoint move backward. Another change introduced a sparse cache of historical offset syncs to translate older positions.

Historical translation then required knowledge of previously emitted checkpoints to prevent unwanted backward movement. Keeping that checkpoint knowledge only inside the current task meant restart lost information needed to make the decision. The conservative response restricted which offset-sync history could be used.

The issue proposes reading checkpoints emitted by previous task generations. That gives the replacement task evidence about the positions already published, so it can preserve the intended progress constraints while allowing translation farther back in the record history.

Startup loads checkpoint state before using it

The 3.7.1 task implementation creates a CheckpointStore for the configured consumer groups. Startup loads that store before starting the offset-sync store, and passes the result into the latter's initialization.

The initialization runs asynchronously through the task's scheduler. A source comment explains that loading can take time and should not block the task's startup method, since a task cannot be stopped until startup completes. This connects state recovery with the connector lifecycle as well as translation correctness.

Polling checks whether the checkpoint store has initialized before returning checkpoint records. The task therefore does not simply create a new empty object and behave as if recovery were complete. Its ability to produce useful checkpoint output depends on the recovered store being ready.

When examining translated candidates, the implementation filters out positions that cannot be translated accurately and applies its checkpoint-recency test. The store participates in that test, replacing reliance on knowledge held only by the current task generation.

The task updates its checkpoint store with the new group checkpoints before turning them into source records for output. It also closes the checkpoint store during shutdown alongside the offset-sync store and clients. Recovery, subsequent updates and cleanup therefore belong to the same task lifecycle. The store is a continuing source of comparison state, rather than a one-time snapshot read and then ignored after startup.

A deliberate upstream rewind is a separate case

The recency logic does not define monotonicity as an unconditional ban on every earlier source position. It explicitly allows a checkpoint after the upstream consumer group rewinds. That is a different situation from accidentally producing an older downstream translation because task history was unavailable.

Otherwise, the code permits an advanced downstream offset and skips repeated or backward downstream checkpoints. The conditions express a concrete policy about progress. Recovering earlier checkpoint state supplies the comparison point needed to apply that policy after restart.

This distinction matters for replay. A user can deliberately move a source consumer backward, while the replication task still needs to avoid unintended rewinds caused by its own recovery. Treating both as one generic monotonicity requirement would obscure the behavior the code actually implements.

Checkpoint availability is distinct from copying progress

The release also includes KAFKA-16622, whose report describes a delayed first checkpoint until a consumer group had caught up. That issue concerns when a usable checkpoint first appears. KAFKA-15905 concerns what a restarted task remembers and which earlier positions it can translate.

I find the separation useful because a running MirrorMaker process alone does not establish useful failover information. Record copying, recovered translation state and emitted checkpoints describe related but distinct progress. The restart repair makes the task recover the historical evidence needed for its checkpoint decisions instead of restricting that evidence to one process lifetime.