Kafka 4.0 requires a completed move away from ZooKeeper

Kafka 4.0 removes ZooKeeper mode rather than performing a ZooKeeper migration during the upgrade. A cluster must already use KRaft before its brokers can move to the new major release. That prerequisite makes the end of ZooKeeper support an operational boundary, not just a reduction in the number of processes a new installation starts.

The March 18 announcement makes 4.0 the first Kafka major release operating entirely without ZooKeeper. The upgrade instructions shipped with 4.0.0 state the required sequence. ZooKeeper clusters must migrate to KRaft first, and the supported KRaft upgrade path begins with software and metadata versions at least 3.3.x.

I value Kafka's ability to retain a stream as history for independent consumers. That data model remains useful through the release. The changes here concern how the system manages its own metadata and how consumer groups decide who reads each partition.

Removing a dependency also removes a deployment mode

The announcement thanks ZooKeeper for more than a decade as Kafka's metadata dependency. Its removal consolidates Kafka around KRaft, but an existing deployment cannot treat that consolidation as an ordinary binary replacement while leaving its old coordination mode intact.

The versioned upgrade guide distinguishes clients from servers. Clients can be upgraded one at a time, subject to the compatibility requirements. Brokers require the KRaft prerequisite and a supported software and metadata baseline.

For older KRaft clusters, the guide recommends moving to 3.9.x before 4.0.x. For ZooKeeper clusters, the necessary operation is migration to KRaft. These are different starting conditions and therefore different routes to the same target release.

That distinction is useful because a version number alone does not describe a cluster's readiness. A cluster's coordination mode and metadata version matter alongside its installed broker binaries. Removing ZooKeeper from a configuration file is not evidence that the metadata migration has occurred.

After rolling the broker binaries, the guide describes finalizing the upgrade through the feature tool. The command uses a local bootstrap address as an example.

bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --release-version 4.0

Finalization is separate from replacing executables. The guide warns that the metadata changes in this release prevent metadata downgrade across that boundary. That gives the feature operation a concrete significance that a generic instruction to restart brokers would miss.

The operational consequence is a sequence with identifiable states. Brokers can run updated code before the metadata upgrade is finalized. Once finalized, the cluster has adopted metadata changes with their own compatibility implications.

Consumer coordination changes independently

Kafka 4.0 also makes the next-generation consumer rebalance protocol generally available. The announcement identifies KIP-848 and says the server side enables the protocol by default. Consumers still opt in explicitly.

group.protocol=consumer

That setting identifies the protocol used by the consumer. Upgrading a broker does not imply that every existing application has changed its group behavior. The distinction lets the cluster support the new protocol while applications choose when to use it.

The KIP-848 design describes why the old arrangement was difficult to operate. Assignment logic and protocol state lived heavily in clients, and group-wide synchronization barriers let one delayed member disturb the rest of the group.

The new design moves coordination toward the broker and lets members converge on assignments independently. The coordinator manages partition handoff dependencies. Its heartbeat protocol carries state used to reconcile the members with the desired assignment.

That is more specific than simply making a rebalance faster. It changes where the system owns assignment progress and how broadly one member's transition needs to synchronize with others. The architectural aim is to reduce group-wide disruption when only part of the assignment changes.

The release announcement limits the GA claim to the consumer protocol. It should not be expanded into a claim that every proposed use of the protocol by every Kafka component has shipped. A design proposal can describe more than the implementation available in one release.

For an application, opting in is meaningful because the coordinator relationship changes. It is not merely a new label for the same client-side protocol. The server becomes responsible for more of the group state that previously depended on clients coordinating through the older mechanism.

Share groups address a different consumption model

Queues for Kafka arrives in early access through KIP-932. The announcement is careful about the terminology. It introduces share groups and cooperative consumption over ordinary Kafka topics, rather than adding a separate queue storage object.

That distinction preserves the topic as the underlying log. The new group type changes how work is distributed to consumers. The announcement compares it with a durable shared subscription, which is a useful description of the intended use without claiming that Kafka has abandoned retention and replay.

I consider partition ownership a central design decision in a Kafka application. Share groups are interesting because they explore a different relationship between the consumers and work in the topic. The early-access label means that exploration has a narrower readiness claim than the GA consumer rebalance protocol beside it.

The two group changes should therefore remain separate in a release assessment. KIP-848 changes consumer group coordination. KIP-932 introduces a group type for queue-like use cases. Their presence in the same release does not give them identical maturity or identical application semantics.

Administrative tooling also changes to recognize the group types. The release adds a general groups command and updates consumer and share group tools. That is necessary because an administration interface that assumes only the older group model can give incomplete or misleading results about newer groups.

Transactions and leadership receive their own changes

The release completes a phase of KIP-890's server-side transaction defenses. The versioned upgrade guide describes 4.0 producer clients advancing the producer epoch for every transaction under the strengthened protocol. This aims to keep duplicate or unintended messages from becoming part of a subsequent transaction.

That concerns the identity of a transaction across producer failure. It is separate from which consumer owns a partition and separate from the controller metadata mode. Keeping those mechanisms distinct makes it easier to understand which component a change protects.

Eligible Leader Replicas also appears in preview. The versioned upgrade guide describes tracking replicas outside the in-sync replica set that can still safely become leader without data loss. The controller retains that eligibility in partition metadata.

The preview status matters. It identifies a different maturity level from the core KRaft-only requirement or the GA consumer protocol. The release combines established changes and evaluation features without making one readiness promise for all of them.

Client and runtime baselines move too

Kafka 4.0 removes old protocol API versions. The announcement says brokers should be at least 2.1 before Java clients move to 4.0, and Java clients should be at least 2.1 before brokers move to 4.0. Non-Java clients need their own compatibility assessment against the removed protocol versions.

This is independent of the Java runtime requirement. Clients and Kafka Streams require Java 11, while brokers, Connect, and tools require Java 17. A compatible network protocol does not establish that the installed Java runtime can execute the component.

The producer's default linger.ms also changes from zero to five milliseconds in the versioned guide. Kafka attributes that choice to batching efficiency, which can offset the additional waiting interval. It is a default change with a mechanism, not a guarantee of lower latency for every producer workload.

Configuration layout reflects the removal of the old deployment choice. The guide says default KRaft properties files move out of the separate config/kraft directory into config. Automation referring to the older path therefore has a file-layout change to handle even when its actual coordination mode is already correct. The directory reorganization follows the consolidation of supported server modes.

The guide also removes old message formats and deprecated interfaces. For example, consumer code using poll(long) must move to poll(Duration), whose timeout behavior differs when awaiting assignment. That illustrates why a major-version application upgrade can require source-level attention even after network compatibility is satisfied.

Kafka 4.0 therefore has several concrete adoption boundaries. The cluster must complete its move to KRaft before upgrading. Applications choose the new consumer protocol independently, while share groups remain early access. Metadata finalization and the updated client and runtime baselines determine which compatibility transitions the deployment has actually completed.