Kafka 1.0 isolates failed log directories within a broker

A broker with several disks has more than one possible failure boundary. Losing a disk need not make the logs on every other disk unusable, but the software has to represent partial storage failure explicitly. Kafka 1.0 adds that representation. A failed log directory can take its replicas offline while the broker continues serving logs on functioning storage.

The 1 November release entry identifies the change with KIP-112. The release notes list disk-failure handling alongside additional health metrics and improvements to authentication errors. These changes are useful together because a broker that survives a narrower failure needs to expose the part that stopped working.

I value Kafka's replicated log as durable history that consumers can revisit. That makes the behavior after storage failure more consequential than the process remaining alive. A running broker is useful only if the system can identify which replicas it can still serve and which copies must be supplied elsewhere.

Failure is associated with a log directory

The replica manager implementation at 1.0.0 handles failure for a specific directory. It first checks whether that directory is already offline. This prevents processing an already handled failure as though it were a new transition.

The method identifies local replicas whose logs belong to the affected directory. It marks their partitions offline in the broker's partition map and removes their partition metrics. The selection is based on the replica's log location, rather than treating every partition on the broker as affected.

That is the mechanism behind partial broker survival. Partitions on other directories remain outside this selected set. The broker does not have to discard its ability to serve healthy logs merely because one storage location can no longer support its replicas.

The distinction is especially relevant when a broker uses several independent disks. A single process can own logs on several storage devices, but the process boundary is larger than any one of those devices. Software that cannot express a directory-level failure tends to turn that smaller failure into a broker-wide event.

Replication work must stop using the failed storage

Marking state is not enough. The handler removes replica fetchers for the affected partitions and removes the directory from the high-watermark checkpoint collection. Continuing those operations against failed storage would leave background work using a resource that the broker has already declared unavailable.

The method then asks the log manager to handle the directory failure and propagates a log-directory event through ZooKeeper. This connects the local storage transition to the wider cluster's coordination. A failure confined to one broker's directory still affects which copies of partition data the cluster can use.

The implementation logs the selected partitions and directory as part of the transition. These details help distinguish the storage location from the set of logical partitions affected by it. A disk problem can involve several replicas, while another partition on the same broker can remain healthy.

Other code in the same source returns a storage error for an offline partition during a leader-epoch request. That gives a caller an explicit representation of the unavailable local replica. It does not pretend that a running broker process makes every partition request serviceable.

Remaining replicas still determine availability

A narrower local failure does not create a replacement copy of the lost data. The cluster still depends on the surviving replicas and their state. The change reduces unnecessary disruption to unrelated logs while preserving the need for a valid copy of each affected partition.

Kafka's design documentation defines replication around individual topic partitions. Leaders and followers maintain copies of that partition's log, with in-sync status determining which replicas satisfy the replication model. The unit affected by a directory failure must therefore be related back to those partition-level copies.

The same documentation distinguishes replication count from minimum in-sync requirements for acknowledged writes. A cluster can still have a running broker while a partition no longer has enough in-sync replicas for its selected write policy. Narrower failure handling improves containment without removing that durability and availability tradeoff.

This is the practical reason to describe the improvement precisely. Kafka 1.0 can keep healthy storage useful when another directory fails. It does not promise that every partition remains available under any combination of storage failures, replication factors and acknowledgment requirements.

Broker liveness is an incomplete health signal

The operations documentation at 1.0.0 includes OfflineLogDirectoryCount under log-manager metrics. It gives an expected value of zero. This metric exposes a condition that a process-only health check could miss after the new failure handling keeps the broker alive.

The same documentation includes UnderMinIsrPartitionCount. That measures partitions whose in-sync replica set falls below the configured minimum. It answers a different question from whether every configured replica is currently in sync.

An under-replicated partition has fewer in-sync copies than its complete replica set. A partition below the minimum crosses the separately configured threshold. These conditions can overlap, but their operational meanings differ because the minimum relates to the selected acknowledgment policy.

A monitor therefore gains a more accurate vocabulary for degraded operation. It can identify offline storage while separately observing replication sufficiency. The broker's existence as a running process remains useful information, but it no longer summarizes all the failure states the implementation can survive.

I prefer these specific signals because they match the resources involved. A directory-level failure deserves a directory-level metric. A partition's replica condition deserves a partition-related metric. That makes the observable state correspond to the smaller failure boundary introduced by the implementation.

The implementation's removal of per-partition metrics during the transition also affects interpretation. An unavailable replica does not necessarily continue reporting its previous measurements as though nothing changed. A monitoring system needs the aggregate storage-failure indication alongside whatever per-partition series remain present. The distinction prevents absence of a particular series from being confused with proof that its last reported state is still current.

Authentication failures also become more specific

The release notes improve SASL authentication diagnostics and treat authentication failures as non-retriable client exceptions. They also include corresponding treatment of SSL authentication failures. These changes distinguish a rejected identity or invalid authentication relationship from an ordinary transient broker connection problem.

This matters operationally because retrying indefinitely cannot correct every authentication failure. A client needs enough information to report that it cannot establish the required identity, rather than presenting every failure as a broker that might recover if contacted again.

The same principle appears in disk handling. A generic process failure loses information about the resource that caused the problem. A specific state or error lets the rest of the system react to the actual condition. The mechanisms differ, but both reduce the need to infer cause from one broad symptom.

The release list also mentions handling authentication failures in transactional producers and the administrative client. Those callers need the error classification as well. A correction in a shared networking component is useful only when the client using it preserves the information in its own control flow.

The version milestone includes concrete client changes

The release entry also documents a relaxation of the idempotent producer's in-flight request constraint. Kafka 0.11 required one in-flight request per connection for this mode. Kafka 1.0 permits up to five, reducing a limit on request concurrency while retaining the idempotent producer model.

That change should remain associated with its version. An application using the initial 0.11 implementation cannot assume the later limit simply because both versions support idempotence. Compatibility descriptions need the configuration rules of the actual client in use.

Java 9 support also arrives in this release, with the project describing faster TLS and CRC32C implementations in that environment. Those implementation improvements are separate from the storage failure model. They should not become a general claim that every unchanged Kafka deployment receives the same performance increase.

Kafka 1.0's operational contribution is concrete. The replica manager can mark a failed directory's partitions offline, stop their associated fetch work and notify coordination while other logs remain usable. Metrics expose the resulting partial failure and its replication consequences. A broker can continue running with less disruption, and the system gives operators evidence that its storage still needs attention.