Kafka 3.5 deprecates ZooKeeper while migration remains early access

Deprecating an old operating mode and making its replacement migration ready for production are separate decisions. The Kafka 3.5 announcement on June 15 makes both statuses explicit. ZooKeeper is deprecated, with removal planned for Kafka 4.0. Migration from an existing ZooKeeper cluster to KRaft remains early access for non-production testing. The combination gives operators a direction without declaring the transition complete.

The distinction is important for an existing deployment. A new installation can choose its metadata arrangement at creation. An installed cluster has state and identities whose continuity matters during migration. A deprecation notice communicates where the project intends to go, while the migration's maturity communicates which path can be relied on today.

I find Kafka's retained history valuable because applications can return to it and maintain independent positions. Changing the metadata system should preserve the cluster that supplies that history. The June release matters partly because it makes the target operating model clearer while continuing to identify the transition as a testing feature.

KRaft assigns controller roles explicitly

The operations documentation shipped with 3.5.0 describes process.roles as the choice between a broker, a controller or a combined server. With no role setting, the server uses ZooKeeper mode. In KRaft, selected controller servers participate in the metadata quorum, with an active controller and hot standbys.

This makes controller placement an explicit deployment decision. The documentation distinguishes dedicated controllers from combined broker-controller processes. Combined mode can simplify a development environment, but it prevents independent rolling or scaling of controllers and brokers. That is an operational coupling with consequences beyond the number of processes deployed.

Separating the roles permits separate maintenance of the controller group and broker group. It also requires an operator to describe how brokers locate the controllers. The quorum-voter configuration supplies controller IDs with their hostnames and ports. Node IDs must be unique across the cluster, regardless of role.

These identities are cluster configuration, not application partition keys. Confusing the two would mix the control-plane membership decision with the application's distribution of records. The operator establishes which processes coordinate metadata, while application and topic configuration describe a different set of responsibilities.

Quorum availability has concrete requirements

The same operations documentation says a majority of controllers must remain alive for metadata-quorum availability. It gives examples with three and five controllers. Those examples distinguish tolerance of one or two controller failures respectively. The number is meaningful because the quorum needs a majority, not merely one surviving process.

Storage initialization is explicit as well. The tools generate a cluster ID and format server storage with that identity. Kafka's documentation explains why automatically formatting empty directories could conceal an error. If a majority of controllers starts with empty metadata logs, a newly elected leader could lack committed information.

That explanation gives the formatting step a correctness purpose. An empty data directory is not automatically evidence of a new cluster. It may indicate that expected storage is absent. Requiring explicit initialization makes that ambiguity visible to the operator before a process treats the directory as authoritative.

A command that reports metadata-quorum state provides a more relevant view than process counts alone. The documentation's example reports the leader and high watermark alongside follower lag. These values describe the replicated metadata partition's progress. A controller process being present is only one prerequisite for that progress.

The documented status command takes a broker bootstrap address. In this illustrative invocation, the hostname and port must identify a reachable broker in the cluster being inspected. It asks for status rather than changing the metadata version or formatting storage.

bin/kafka-metadata-quorum.sh --bootstrap-server broker.example.com:9092 describe --status

The returned voter list can be compared with the configured controller membership. Its purpose is to examine the running quorum that the configuration was intended to establish.

Deprecation does not migrate existing state

The release adds a storage-tool mechanism for configuring SCRAM in KRaft broker authentication. This is part of making the new operating mode usable with requirements that existing installations already have. Supporting a new cluster's authentication arrangement and converting an old cluster's metadata remain different problems.

Kafka's 3.5.0 upgrade instructions also separate rolling out software from enabling a newer protocol or metadata version. A package installation does not by itself communicate every compatibility transition the cluster can make. This distinction becomes more important when the project supports more than one metadata mode.

For an existing ZooKeeper deployment, the deprecation therefore changes planning assumptions. It does not remove ZooKeeper from this release or automatically move its metadata. The early-access label continues to define the supported context for trying that migration. Reading the two statements together gives a more accurate account than treating either one as the entire release policy.

I prefer that explicit staging because it allows the installed cluster's requirements to remain part of the discussion. A target design is useful, but the transition must account for the system already storing the records. June's announcement identifies the destination and the present maturity of the route separately.

Consumer assignment now uses rack information

The release has another change whose usefulness depends on the distinction between a protocol and its implementation. Kafka 3.4 included the protocol changes for rack-aware consumer assignment. Kafka 3.5 updates built-in assignors to use that information. A field available in the protocol becomes actionable behavior in the supplied client implementation.

The RangeAssignor source at the 3.5.0 tag documents the policy. It tries to align consumer racks with partition-replica racks on a best-effort basis. Balanced assignment takes priority over rack alignment. For topics that need co-partitioning, that relationship also takes precedence.

That ordering matters because several desirable properties can conflict. Moving a partition to match a rack can change the distribution of assignments. Keeping corresponding partitions together may limit which consumers can receive them. The implementation must choose among those goals, and its documented priorities explain why locality is not an unconditional rule.

The source builds its rack information from partition replicas. It considers replicas outside the in-sync set as well, on the expectation that they may return. The comments explain that consumer groups do not rebalance merely because the in-sync set changes. The assignment therefore describes a more persistent locality choice than the exact set of replicas caught up at one instant.

Locality is part of assignment policy

The consumer configuration source exposes the client-rack configuration used by this arrangement. A configured location gives the assignor information it can compare with broker-replica locations. Without meaningful locations on both sides, a locality preference has no useful correspondence to establish.

This is different from changing the application key. A key helps determine which partition receives a record. A consumer assignment determines which group member handles that partition. Rack awareness can influence the latter without redefining the ordering relationship the application chose by partitioning its events.

The distinction is useful when explaining the limits of the optimization. Locality can improve the placement of a consumer relative to available replicas. It does not create global ordering or change the fact that a partition is the unit being assigned. The application still needs to choose a key appropriate to the events it expects to process together.

The implementation retains range assignment for remaining partitions after rack matching. Its comments explicitly allow those assignments to be misaligned with rack locations. A successful rack-aware assignment process therefore does not mean every partition has a local match. The result reflects the documented balance and co-partitioning constraints as well as the available placements.

The release advances two operating concerns

The control-plane changes make KRaft's deployment responsibilities more explicit while marking ZooKeeper deprecated. The consumer changes make location information useful in the supplied assignment implementations. Both affect how a running Kafka system is arranged, though one concerns metadata authority and the other concerns consumers relative to replicas.

For an existing installation, the immediate result is a clearer set of separate decisions. Controller roles and storage identities describe the KRaft environment. The migration retains its early-access scope, while consumer assignment can now use rack information according to the built-in policy's priorities. A locality preference remains subordinate to the assignment constraints the implementation documents.