Kafka Cluster Linking preserves partition offsets across clusters

Moving a Kafka workload involves moving the meaning of its offsets. A consumer position identifies a place in a particular partition log. Copying the same business events into another topic does not, by itself, establish that position in the destination. A migration therefore needs an account of where consumption can resume, as well as an account of which records have arrived.

Confluent's October 5 introduction of Cluster Linking addresses that problem through broker-level replication. The preview accompanies Confluent Platform 6.0, with early access in selected Confluent Cloud regions. A destination partition leader fetches its source partition into a read-only mirror topic. Mirroring preserves partition identity and offsets, while the clusters retain independent replication groups. The destination requires Confluent Server.

I find the offset preservation more consequential than the reduction in replication infrastructure. Kafka's usefulness comes from retaining a history that independent consumers can revisit. Moving that history becomes easier to reason about when an event retains its location within the partition.

A copy with a defined relationship to the original

Apache Kafka's 2.6 replication design already describes followers pulling partition logs from their leader. Those replicas preserve message order and offsets. Cluster Linking applies a related fetch mechanism across a cluster boundary, with a different destination role. The remote copy belongs to its own cluster and has its own local replication responsibilities.

Consider an illustrative source partition containing an order event at offset 417. A consumer that has processed the event can record its next position as 418. If another system merely republishes the order into a fresh partition, its new offset depends on that destination's log. The value 418 then needs translation before it can describe the corresponding place. An offset-preserving mirror removes that particular translation problem.

The example says nothing about whether the consumer's committed position has also moved. An application needs its progress information as well as its records. Preserving the address space makes the two compatible, but it does not prove that a particular application has resumed at the right point. Consumer group state and application state remain separate subjects in a migration plan.

This distinction matters for applications that maintain external results. A database update may already reflect an event whose processing position has not been durably recorded. Resuming from an earlier point can repeat that work. Conversely, choosing a later position without a corresponding result can omit work. These are application recovery questions, even when the destination has an exact copy of the partition history.

The mirror's read-only status also has a practical meaning. An independent producer cannot insert another record at the destination while mirroring continues. Allowing unrelated writes would require deciding how they fit into the source's offset sequence. A source-controlled log gives the destination a precise sequence to reproduce, which is the basis for the simpler correspondence between positions.

Independent clusters have independent durability

Cluster Linking replicates asynchronously. Source writes therefore do not wait for a remote cluster to participate in the source's local replication decision. The announcement describes separate in-sync replica sets for the two clusters. A healthy local cluster can continue accepting writes while its remote mirror falls behind.

Kafka's existing replication rules explain why this boundary deserves attention. In the Apache design, acks=all concerns the partition's current in-sync replicas. It does not mean every configured replica has necessarily received the record. The min.insync.replicas setting adds a lower bound for writes that request that acknowledgement. Neither setting, applied at the source, establishes that an asynchronous destination has received the same write.

For a recovery design, a source acknowledgement and a remote recovery point must therefore be described separately. A hypothetical failure can destroy a source after it has acknowledged a record but before the destination has fetched that record. The destination cannot recover data it has never received. Offset preservation identifies the copied records precisely, but cannot remove the interval in which they exist only at the source.

I prefer this explicit separation when evaluating cross-cluster replication. It allows a local write policy to remain understandable without making every remote delay part of local acknowledgement latency. The cost is a recovery objective that depends on the progress of replication between the clusters.

That cost affects the interpretation of lag. A reachable destination is not necessarily current. A consumer able to read its mirror does not demonstrate that the source's latest acknowledged write is present. Recovery decisions need evidence about the replicated end of each relevant partition, because the application may depend on several partitions that have advanced by different amounts.

Promotion changes who controls the log

The preview permits a mirror topic to become an ordinary writable topic. Confluent describes a migration in which replication catches up before clients move to the promoted destination. This changes the destination's role from reproducing another log to accepting new writes of its own.

For a planned move, stopping new source writes makes the replication target finite. Without that step, a destination can reach a recently observed source position while the source continues to advance. An application cutover needs a defined final source position and an agreement about which producers may write next. The offset relationship makes that agreement easier to express, but does not decide producer ownership.

An unplanned recovery has a different constraint because the final source position may be unavailable. Promoting the available copy permits new destination activity, but the application must accept whatever recovery point that copy contains. Planned migration and disaster recovery can use the same copying mechanism while requiring different evidence before the destination becomes authoritative.

The initial preview also has a concrete network direction. Destination brokers initiate the connection to the source. Source-initiated links appear in the announcement as future work, so a design that requires that direction cannot assume the preview provides it. Firewall placement and reachability determine whether the announced mechanism fits a particular deployment.

I see the strongest case here in moving retained partition history while preserving the positions applications use to interpret it. That still leaves application ownership and recovery policy to the system around Kafka. Once a promoted destination starts accepting its own records, that system must have an explicit answer about which cluster now owns the topic's subsequent history.