Kafka 3.7 gives KRaft a view of individual log directories
Kafka 3.7 previews JBOD support in KRaft. Directory identities let the controller distinguish a failed disk from a failed broker and update the affected replica assignments.
A broker can keep sending heartbeats while one of its disks has stopped serving partitions. Broker liveness alone therefore cannot tell a controller whether every replica on that broker remains usable. Supporting several independent log directories requires the metadata system to represent failures smaller than the entire process.
Kafka 3.7.0, announced on February 27, introduces early-access JBOD support in KRaft. The announcement explicitly says this capability is not yet recommended for production. Its importance is the missing directory-level failure information, rather than simply accepting a list of filesystem paths.
Kafka's durable, replicated log is what makes it useful to me. The durability model depends on knowing which replicas can still serve their data. A broker that remains alive while losing a subset of storage needs more precise representation than a single healthy-or-dead status.
A live broker can host unavailable replicas
KIP-858's design explains the gap. In KRaft, brokers establish membership through heartbeats to the active controller. A failed data directory need not stop the broker, so heartbeats alone do not tell the controller to move leadership away from replicas stored there.
The ZooKeeper implementation has a notification path for log-directory failure. The controller can use per-partition responses to determine which replicas are affected. KIP-858 describes a KRaft design that avoids having one failure notification enumerate every partition in the failed directory.
The distinction matters for large brokers. Failure reporting that grows with the number of replicas on a disk can produce large requests precisely when storage is already malfunctioning. Giving directories persistent identities makes it possible to communicate the failed storage unit while retaining replica-to-directory relationships in metadata.
The metadata directory is a separate case in the proposal. If that directory fails, the broker cannot continue, and lost heartbeats communicate the process failure. The JBOD work addresses data-directory failures that leave enough of the broker functioning to continue serving other replicas.
Directory identity survives a path change
The design adds directory.id to each directory's meta.properties. It is a randomly generated UUID, distinct from the broker's node identity and the cluster identity. Storage formatting generates the directory identity, and broker startup reads it.
This makes the storage object identifiable independently of its mount path. A string such as /mnt/d1 tells the operating system where to find a directory in one environment. A persisted UUID lets cluster metadata refer to that directory even when the path itself is not the durable identity.
The proposal includes directory identities in broker registration and adds a parallel directory field to partition metadata. Its ordering follows the replica list, connecting each replica with its hosting directory. The controller can then reason about a replica's availability using both the broker and the directory hosting it.
Special identities represent cases where a directory assignment is unknown, lost or part of a transition from metadata without directory information. Those states are necessary because an upgrade cannot always infer the location of every existing replica before the broker reports it.
Failure reporting joins the heartbeat protocol
KIP-858 adds offline directory information to broker heartbeats. A broker with multiple data directories reports newly failed directories and continues including accumulated failures until restart. The controller can use those identifiers to update leadership and in-sync replica state for the affected replicas.
The design also introduces AssignReplicasToDirs to communicate replica placement. A persistent mapping is useful only if movement and initial placement keep it current. Otherwise, a later directory failure could refer to a stale assignment and affect the wrong interpretation of replica availability.
There is a further failure path when notification itself does not succeed. The proposal introduces log.dir.failure.timeout.ms, with a default of 30 seconds. If a broker cannot communicate a directory failure within that period while still hosting leadership on the failed directory, it shuts down.
That shutdown gives the controller another observable failure signal. A broker cannot continue indefinitely advertising liveness while leadership remains assigned to storage it knows is unavailable. The timeout connects local knowledge of failed storage with the cluster's need to elect usable leaders.
Placement metadata becomes part of storage operation
The design checks placement when a broker catches up with metadata. Existing replicas with unknown directory assignments need their actual directory reported. A newly created replica likewise needs an assignment that the controller can retain.
This is more work than distributing partitions across a collection of paths. The controller's representation must agree with the physical location because that agreement determines what a future directory failure means. The same requirement applies when moving a replica between directories on one broker.
The proposal rejects duplicate directory UUIDs at startup. Reusing one identity for two directories would make the mapping ambiguous before any failure notification arrived.
I find this a useful example of the operational detail hidden by the word replication. Retaining several copies is only one part of availability. The metadata system must also know which copy belongs to failed storage while the rest of its broker remains active.
The release also narrows future compatibility
Kafka 3.7 marks client protocol API versions introduced before Kafka 2.1 as deprecated, with removal planned for 4.0. The announcement says new metrics help identify those requests. This is protocol compatibility, so an application's client behavior matters independently of the broker's own runtime.
Java 11 support for broker and tools modules is also deprecated, with removal planned for 4.0. The announcement distinguishes those modules from clients, Streams and Connect. It would be incorrect to read the broker runtime change as requiring every client application to move to the same Java runtime immediately.
These changes put 3.7 in a transitional role. It gives operators an early view of directory-aware KRaft storage while exposing older protocol use ahead of planned removal. For JBOD, the concrete addition is a metadata model that can identify a failed directory and the replicas assigned to it while the broker itself continues running.