Kafka 2.8 puts the metadata quorum inside Kafka

A Kafka controller needs a durable account of the cluster it controls. Kafka 2.8.0, released April 19, offers early access to keeping that metadata in Kafka's own controller quorum. The mode is called KRaft, short for Kafka Raft metadata mode. ZooKeeper is no longer required for a cluster created in this mode, although the release is explicitly for testing.

The guide shipped with 2.8.0 identifies this as the KIP-500 direction. Its warning is specific. Existing ZooKeeper clusters cannot migrate into the mode, and the early-access cluster has no supported upgrade path. Data loss is a stated possibility. This is an opportunity to examine the new metadata architecture, not a supported conversion of an existing production cluster.

I find the direction interesting because Kafka already makes durable history central to its data model. Bringing cluster metadata under its own replicated log applies a related idea to the system's administration. The operational benefit needs to be understood through the new responsibilities, not merely through the disappearance of one service name.

Controllers form an explicit quorum

In ZooKeeper mode, any Kafka server can become the controller. The KRaft guide instead describes a selected group of controller servers that participate in the metadata quorum. One is active and the others are hot standbys. This makes controller membership a configured part of the deployment.

A majority of that group must remain alive for availability. The guide describes three or five controllers as typical choices. Three can tolerate one controller failure, while five can tolerate two. Those counts describe the quorum's availability condition, independently of how many brokers store a topic's records.

Each server receives a controller.quorum.voters setting. It lists controller node identifiers with hostnames and ports. Every controller must appear in the list, and its configured node.id must match the corresponding identifier. Brokers also receive the voter configuration so they know which controller quorum to use.

Clients do not configure that voter list. The guide says producers, consumers and administrative tools still connect through the broker's configured endpoint. Moving metadata coordination changes server responsibilities while retaining the ordinary client entry point. An application does not need to participate in controller elections merely because the cluster no longer runs ZooKeeper.

Roles can share a process or remain separate

The new process.roles setting selects what a Kafka server does. broker and controller identify separate roles, and broker,controller combines them. Omitting the setting selects ZooKeeper mode. This is an explicit mode choice, with the early-access guide warning that switching modes requires reformatting rather than an ordinary in-place transition.

The combined-node configuration supplied with the release includes these relationships:

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
advertised.listeners=PLAINTEXT://localhost:9092
controller.listener.names=CONTROLLER

This is an excerpt from a single-node example, not a complete resilient deployment. The same process has a broker endpoint and a controller endpoint. The voter identifier points back to that one controller. There is no second node available to preserve this quorum if the process stops.

The complete example also sets inter.broker.listener.name to PLAINTEXT and maps listener names to security protocols. Those settings distinguish the name used to select an endpoint from the protocol used on that endpoint. The controller listener is explicitly named, while the advertised client listener points to the broker port. Keeping those relationships clear helps prevent treating all configured addresses as interchangeable entry points.

Combining the roles makes a simple installation easier and avoids some fixed JVM overhead. The guide names isolation as the tradeoff. If broker activity causes the process to run out of memory, the controller in that process shares the failure. Removing a separate service does not make the resources needed for coordination disappear.

Separate controller processes let the deployment make that distinction explicit. They still need configuration and storage, while their lifecycle is less directly tied to a broker process. The guide does not claim one arrangement is appropriate for every workload. It explains what is gained and what is coupled when the roles share a process.

Formatting is an intentional operation

KRaft introduces an explicit storage preparation step. The guide first generates a cluster identifier with kafka-storage.sh random-uuid. It then formats each node's directories using that identifier and the node's configuration. All nodes in the same cluster must use the same cluster ID.

This differs from automatically formatting blank directories and generating an identifier at startup. The guide gives an operational reason for the change. A data directory can appear blank when its intended filesystem has failed to mount. Automatically initializing it would turn a missing storage device into what looks like a fresh member.

For a metadata quorum, that can be worse than a local startup mistake. If a majority of controllers starts with blank logs, it may elect a leader without the cluster's metadata history. The guide uses two out of three controllers as its example. Requiring explicit formatting prevents an ordinary start from silently treating absent history as permission to initialize new state.

I regard that decision as one of the most useful details in this preview. Convenience during first startup and safety during recovery have different requirements. An operator should be able to distinguish creating a new metadata history from reopening an existing one, especially when a missing mount can make the directories look similar.

The cluster ID gives the format operation a shared identity as well as a filesystem effect. It is supplied deliberately across the participating nodes. That makes initialization something the operator can describe and repeat with the intended membership, rather than a decision each process independently makes when its directory is empty.

Metadata history is available for inspection

The guide includes a decoder mode for kafka-dump-log.sh that reads the metadata log. Its example contains broker registration records, fencing and unfencing records, and topic and partition records. These are concrete entries in the history that describes the cluster. They are distinct from application messages published to a user topic.

The metadata shell provides another view. Its example navigates through topics and reads a partition's metadata, including replicas and in-sync replicas. A developer examining the preview can connect the configured cluster state with the records representing it. This is useful evidence about how the new mode organizes information, without needing to infer the representation from broker logs alone.

The distinction between history and a current view is also relevant to restart behavior. The early-access guide says metadata snapshots under KIP-630 are not yet supported. Without that work, restart time can grow as the metadata log grows. Eliminating ZooKeeper therefore does not eliminate recovery work, and the preview documentation identifies where that work remains incomplete.

The preview has a deliberately limited scope

Transactions and exactly-once semantics are among the features the guide says are not fully implemented in KRaft mode. It also lists partition reassignment and adding partitions to existing topics as incomplete. Some security functions, including configuring an authorizer and setting up SCRAM, remain missing. These are operational capabilities that an ordinary Kafka deployment may rely on.

The supplied combined configuration further reflects its development purpose. Internal offsets and transaction-state replication factors are set to one. Its comments recommend larger values for uses beyond development testing. Those values are part of the example's assumptions, not a statement that moving metadata into Kafka removes the need for replicated internal state.

The example stores logs under /tmp/kraft-combined-logs. That path also appears in the guide's formatting and inspection examples, connecting the storage configuration to the files the tools examine. An experiment using another directory needs to preserve that relationship instead of inspecting a path left over from a different run.

The release can still be valuable within that scope. It exposes the server roles and the initialization model early enough for users to study them. The selected controller group, its shared identity and its explicit storage history are observable parts of the new architecture. They explain what running without ZooKeeper actually means in the software available today.

Kafka 2.8's early access makes the metadata quorum part of Kafka's own deployment. Brokers know the selected controllers, controllers retain the metadata log, and startup depends on deliberately formatted storage. Those are the mechanisms behind the simpler service count, with the documented feature and recovery gaps still defining where this preview can be used.