Kafka 3.3.1 makes KRaft a production choice for new clusters

Kafka can now manage its own production metadata for a new cluster, but that does not turn an existing ZooKeeper deployment into a KRaft cluster. The October 3 release summary marks KRaft production-ready through KIP-833. Kafka 3.3.1 is the announced release carrying that milestone. The project advises against 3.3.0 after finding a significant defect before its release announcement.

The operational change is a dedicated metadata quorum maintained by Kafka itself. Controllers have explicit roles, metadata storage receives deliberate initialization, and the distribution includes tools for inspecting quorum state. Removing the external coordination dependency gives Kafka responsibility for those tasks inside its own operating model.

I value Kafka's replicated log and the way retained history separates producers from independent consumers. Self-managed metadata applies a related model to the cluster's own state. Its appeal depends on making the new administrative responsibilities clear enough to operate without treating the removal of ZooKeeper as the whole design.

Controller and broker roles are explicit

The 3.3.1 operations guide describes process.roles. A server can be a broker, a controller or both. Omitting the property leaves the server in ZooKeeper mode. KRaft is therefore an explicitly configured execution mode, not an automatic consequence of installing the new binaries.

Combined servers run both roles in one process arrangement. The guide describes them as convenient for development and small use cases, while advising against combined mode for critical deployments. Combining roles prevents controllers from being rolled or scaled independently of brokers. It also reduces isolation between the controller and the rest of the workload.

That is a concrete topology tradeoff. Reducing the number of separate processes can simplify a local environment. Separating roles provides independent lifecycle control in a production design. The production-ready milestone does not erase that distinction or recommend every topology supported by the configuration syntax.

The guide also changes how controller membership is understood. Specific servers are designated as KRaft controllers. Each participates in the metadata quorum as the active controller or a standby. That differs from an arrangement in which any broker is eligible to take the controller role.

Voter identity belongs in every node's configuration

The controller.quorum.voters property identifies the quorum members through node IDs and network endpoints. Every broker and controller needs that list. Each ID must match the configured ID of the corresponding controller, and node IDs must be unique across all servers regardless of role.

An illustrative configuration fragment is the following.

process.roles=controller
node.id=1
controller.quorum.voters=1@controller1.example.com:9093,2@controller2.example.com:9093,3@controller3.example.com:9093

This fragment identifies a role and quorum membership. It omits listener security and the rest of the server configuration. Its purpose is to show that the controller's local identity and the shared voter list must agree. Copying a fragment while leaving duplicate node IDs would contradict that requirement.

The operations guide explains the majority requirement for availability. A three-controller quorum can tolerate one failed controller, while five can tolerate two. Its deployment guidance nevertheless recommends three controllers for critical environments in this release because a rare partial-network-failure limitation can make larger quorums unavailable. The generic majority arithmetic and the release-specific recommendation need to be read together.

Formatting storage becomes deliberate

The storage tool generates a cluster ID with kafka-storage.sh random-uuid. That same ID must be used when formatting the servers in the new cluster. The guide explains why KRaft no longer silently formats empty storage directories and generates an ID during startup.

Automatic initialization can hide a failure in which a node unexpectedly starts against an empty directory. For metadata, the consequence is serious. If a majority of controllers starts with empty logs, an elected leader can lack previously committed state. Requiring explicit formatting separates a new cluster from an existing cluster whose data is missing.

This is an operational safeguard with a specific purpose. An empty directory does not carry enough information to tell whether initialization is intended. A separate formatting step makes the administrator supply that intent before the server treats the directory as its metadata store.

My preference is for initialization to be explicit when losing history would change the meaning of the system. The extra command is justified by the ambiguity it removes. A successful process start should not conceal that the process has started with the wrong state.

Quorum inspection becomes a normal administrative task

The distribution includes kafka-metadata-quorum.sh to inspect the metadata partition. The operations guide demonstrates a status summary through a broker endpoint.

bin/kafka-metadata-quorum.sh --bootstrap-server broker.example.com:9092 describe --status

The documented output includes cluster ID, leader ID and epoch, high watermark, and follower-lag information. It also identifies voters and observers. Those fields describe the metadata quorum rather than an application's consumer-group progress. Both involve offsets and lag, but they refer to different replicated state.

The guide also documents describe --replication for controller recovery. It exposes each node's log-end offset, lag and fetch timestamps. Before replacing lost controller storage, the administrator needs evidence that a majority retains the committed data. The recovery procedure ties that decision to the quorum's actual replication state.

This is a new responsibility to understand, even if the organization no longer maintains a ZooKeeper ensemble. A majority of processes being reachable does not by itself describe whether those processes retain the required metadata history. The inspection tools make the distinction observable.

Recovery preserves the committed metadata history

Controllers store metadata in metadata.log.dir, or the first log directory when that property is unset. The recovery guidance says a replacement controller should not be formatted and started until a majority of controllers has the committed data. It recommends comparing lag and the last-fetch and last-caught-up timestamps to establish progress.

The guide allows --ignore-formatted only for a specific combined-mode case where metadata storage was lost but the other log directories remain formatted. That qualification matters because a convenience flag is not a general instruction to override storage-state errors. The state of each directory determines what the recovery command is supposed to accomplish.

Metadata decoding tools provide another form of inspection. kafka-dump-log.sh can decode metadata log segments and snapshots, and kafka-metadata-shell.sh offers an interactive view of stored state. These tools make the cluster's administrative history inspectable through its persisted representation.

The shell example in the guide lists topics and reads a partition's metadata. Its fields include replica assignments, the in-sync replica set and the partition leader, along with leader and partition epochs. That view describes how the cluster organizes an application partition. It does not print the application's records. Distinguishing the metadata partition from the user partition it describes is necessary when interpreting both kinds of log through the tools supplied by the distribution.

Binary upgrades and metadata-version changes are separate

The 3.3.1 upgrade documentation distinguishes installing newer code from advancing metadata.version in a KRaft cluster. Once the metadata version advances to the production format, it cannot be lowered to the pre-production 3.0, 3.1 or 3.2 metadata versions.

That boundary is significant for an existing experimental KRaft installation. Running newer code and committing to a newer metadata representation are different stages. A rollback plan must account for the persisted metadata version, not merely the availability of the older executable.

The release summary names KIP-778 for KRaft upgrades and KIP-835 for quorum monitoring. Those capabilities accompany production readiness because a metadata implementation needs a supported lifecycle after its initial creation. Starting a new cluster is only the first operation it must sustain.

New clusters have a different decision from existing ones

The operations guide lists features not fully implemented in KRaft, including administrative SCRAM-user configuration, multi-directory JBOD, delegation tokens and migration from ZooKeeper. These are explicit limits on the available mode. An organization relying on one of those capabilities cannot infer support from the production-ready label alone.

Existing ZooKeeper clusters consequently have a different choice from a new installation. This release does not provide their conversion procedure. The upgrade notes describe KRaft readiness specifically for new clusters, which preserves that boundary without predicting a date for a later migration implementation.

I regard the milestone as meaningful because the supported operating model now includes explicit roles and inspectable metadata storage. The immediate production choice is whether a new cluster's requirements fit the documented KRaft capabilities and topology. Existing ZooKeeper clusters continue to require their current metadata arrangement until a migration capability exists.