Kafka 3.4 previews migration of existing ZooKeeper metadata

Starting a new cluster with a different metadata system is easier than changing the authority of an existing cluster. Existing brokers already have registrations and partition assignments that must continue to mean the same thing after the transition. Kafka 3.4, released February 7, introduces an early-access path for migrating ZooKeeper metadata to KRaft. The change addresses the installed cluster, which cannot obtain its current state by behaving like a new deployment.

The release announcement makes the scope explicit. Migration is for non-production testing in this release. ACL migration and downgrading to ZooKeeper during or after migration are unsupported. Those limits matter when interpreting the broader objective of a migration mechanism. A design intended to ease eventual transition does not establish that every recovery route is available in this implementation.

I value Kafka's retained history and the ability of consumers to maintain independent positions in it. Replacing the metadata system should preserve the cluster that gives that history meaning. The important engineering problem is continuity of authority and identity, not whether a different control-plane process can be started.

The driver represents a sequence of states

The migration driver in the 3.4.0 source uses an event queue to serialize migration events. It maintains explicit states for waiting on the controller quorum and brokers, becoming controller, and migrating ZooKeeper metadata. Later states cover communication with brokers and dual writes.

That organization explains why the operation is not one configuration substitution. The destination controller has to know which source brokers are ready. Existing metadata must reach the new log before subsequent work can rely on it. The driver has to distinguish those conditions even when several other threads produce the events that report them.

Serialization of the migration events gives the state transitions a defined place to happen. The code checks whether a proposed state change is valid and logs transitions. That makes migration progress different from a collection of independent startup successes. Every process can be running while the driver still waits for a prerequisite.

This is useful operationally because a migration needs an account of its current phase. A stalled broker-registration phase and a failed metadata-copy phase require different investigation. Treating both as a cluster that has not finished starting would hide the specific dependency blocking progress. The explicit states provide names for those distinctions.

Broker registration establishes readiness

The driver compares broker IDs known through ZooKeeper topic assignments with ZooKeeper-mode brokers registered in the KRaft metadata image. If some expected brokers have not registered, it waits and reports their IDs. This connects the migration's readiness condition to the cluster that already exists, rather than to an arbitrary count of newly started processes.

That comparison matters because a destination controller must account for existing ownership. A broker that has not completed the relevant registration is different from a broker absent from an empty test cluster. The source metadata gives the driver evidence about which IDs it expects to see. Registration supplies evidence that those brokers have reached the migration-aware state.

Identity therefore becomes part of the transition contract. A node's successful launch alone does not establish that the new metadata system associates it with the existing assignment. The registration comparison explains why an apparently healthy process can still be a missing prerequisite from the driver's perspective. This is a mechanism visible in the tagged code, without relying on later migration guidance.

The driver also waits for an initial metadata image before performing that comparison. It cannot use an empty image as though it were an authoritative view of all registrations. That check distinguishes absence of information from information that a broker is absent. The difference is small in code and substantial in a distributed state transition.

Copying metadata establishes a destination position

The migration event begins a migration with the record consumer and reads ZooKeeper metadata in batches. Each batch is accepted into the destination process, and the driver waits for that acceptance to complete. Completing the migration returns a metadata offset and epoch. The driver records that position in its migration leadership state.

An offset and epoch give the subsequent phases a specific position to reference. They describe where the imported metadata stands in the destination log. Merely reporting that some records were read would leave the transition without that relationship. The code's completion message includes the number of generated metadata records and the resulting offset and epoch.

The destination metadata image must catch up to the migration position before the driver sends the next set of controller RPCs to brokers. The comparison prevents the communication phase from relying on an image behind the point established by the copy. It is another example of readiness defined by state rather than by elapsed time.

The imported content is cluster metadata. This is separate from copying all application records into another cluster. Conflating the two would give the wrong explanation of what KRaft replaces. Kafka's migration announcement describes a change to the metadata log and rolling broker conversion, not a new application-history replication procedure.

Dual writes support the mixed period

Once in dual-write state, the driver handles metadata deltas and writes corresponding changes to ZooKeeper. The tagged implementation includes handling for topics and their partitions, alongside configurations and client quotas. It also propagates controller information to the brokers using a legacy propagation interface.

That behavior explains how the new authority can coexist with brokers still using the older metadata arrangement during the transition. Imported state is only the beginning. A live cluster can change after the initial copy, so the migration must also account for later updates. Otherwise, the two representations would diverge as soon as an administrative operation changed the cluster.

The code checks the metadata position before applying an update to ZooKeeper. It can ignore an image or delta that represents metadata already written there. This makes progress tracking part of the ongoing synchronization, as well as part of the initial copy. Reprocessing a publication event should not be interpreted without reference to the position already recorded.

Dual writes should not be described as proof of supported rollback in 3.4. The release announcement explicitly excludes that capability. A mechanism can supply some of the state relationships useful to a broader design while the released feature still has narrower supported behavior. The release's limitations determine what an operator can rely on today.

Software upgrade and metadata migration differ

Kafka's tagged upgrade instructions distinguish software rollout from enabling newer protocol or metadata behavior. For ZooKeeper clusters, brokers can run new code while retaining the earlier inter-broker protocol setting during the initial rollout. KRaft clusters use a metadata-version feature for their corresponding compatibility transition.

That separation is relevant to understanding the preview. Installing the new version gives a process the implementation that supports the new work. Changing the cluster's metadata authority is a different operation involving existing state and coordinated transitions. A release that contains migration code does not migrate every cluster when its package is installed.

The controller configuration shipped with 3.4.0 also makes the controller role explicit. It identifies a node ID and controller-quorum voters. Those are configuration objects an operator must understand independently of the application topics the cluster stores. The sample also gives the controller its own listener name and address, making the communication endpoint explicit in the configuration. It is a starting point for a KRaft controller, not a complete production migration procedure.

I regard the early-access release as useful because it exposes the transition machinery for examination on a non-production cluster. Its state progression can be inspected against registrations and metadata positions. That produces more specific feedback than a claim that a cluster either migrated or failed, while respecting the release's unsupported ACL and rollback paths.

For existing ZooKeeper deployments, Kafka 3.4 supplies the first released testing path described by the announcement. The driver accounts for readiness before copying metadata, records the destination position, then coordinates later updates and broker communication. The result to examine is continuity of the existing cluster's metadata through those phases, with production use still outside the stated scope of this release.