Kafka 4.3 can stop placing replicas on retiring storage

Removing partitions from a disk used to make that disk more attractive to Kafka's placement logic. KIP-1066 explains the problem. A broker chooses the log directory with the fewest partitions for new placement, so an operator emptying a directory could see new work arrive there. Broker retirement had a similar race with ongoing topic creation.

Kafka 4.3.0, released on May 22, adds cordoned.log.dirs. A cordoned directory continues operating, but new partitions cannot be allocated to it. Cordoning every directory makes the broker ineligible for new partition placement.

I value Kafka's retained history and independent consumption, but the replicas holding that history still need an orderly maintenance process. This change supplies a missing expression of operator intent. The placement system can now distinguish storage that is healthy from storage that is healthy but scheduled to leave.

Cordoning excludes new placement without moving existing data

The distinction between cordoning and evacuation is central. The configuration stops new assignments to the selected directories. Existing partitions continue to function there. Moving them remains a separate operation, which means the operator can establish the placement restriction before starting reassignment.

KIP-1066 also gives a scale-out example. After adding brokers to a cluster, ordinary round-robin placement still includes the old brokers. Temporarily cordoning old storage can direct new placement toward the added capacity. The capability expresses eligibility, so it does not need to replace the entire placement algorithm with an application-specific scheduler.

The configuration is a list of paths that must appear in the broker's configured log directories. An asterisk selects all of them. It can be set in broker properties or changed dynamically per broker. The proposal gives this form for cordoning one directory on broker zero:

bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config cordoned.log.dirs=/tmp/kraft-broker0-logs0 --entity-type brokers --entity-name 0

The path in that example must correspond to the selected broker's storage. The operation changes the broker configuration. It does not remove a directory or copy any replica data, and successful configuration should not be described as completed decommissioning.

Brokers translate paths into controller-visible identities

The controller does not use a host's filesystem paths as its storage identity. KIP-1066 has brokers send the UUIDs of cordoned directories in heartbeats after catching up with metadata. The controller then persists the cordoned state through metadata records.

The placement logic uses a ClusterDescriber interface. Its usable-broker view excludes a broker with no uncordoned directory. Its default-directory selection chooses an uncordoned directory. The target directory supplied by the controller remains a hint because the broker performs actual directory selection, which must also honor cordoning.

That division explains why the feature needs both broker and controller changes. Restricting only the local directory picker would leave the controller assigning work to an ineligible broker. Restricting only the controller would leave the final local choice able to select a retiring disk.

The proposal gates configuration on a supporting metadata version. Installing binaries alone does not establish that the cluster can record and interpret the new state. The metadata requirement belongs to the operational meaning of the feature, rather than being an incidental command-line restriction.

Persisting the exclusion also matters when a broker is offline. The controller retains the cordoned-directory information in the metadata log, so eligibility does not depend on asking the broker's filesystem at every placement decision. The broker's heartbeat communicates local path decisions as directory identities the cluster can retain and use. This connects a host-level maintenance instruction to durable cluster state.

Placement failures become explicit

Cordoning can leave too few eligible brokers for a requested replication factor. In that case, automatic topic or partition placement returns INVALID_REPLICATION_FACTOR. An explicit assignment naming brokers without eligible directories returns INVALID_REPLICA_ASSIGNMENT.

Reassignment has its own failures. Assigning a replica to a broker with no uncordoned directory returns INELIGIBLE_REPLICA, while targeting a cordoned directory through AlterReplicaLogDirs returns KAFKA_STORAGE_ERROR. These results make placement intent enforceable even when a caller supplies a destination.

Observability accompanies the restriction. The proposal adds isCordoned() to LogDirDescription and includes cordoned state in log-directory descriptions. Metrics report a directory's cordoned status and the broker's count of cordoned directories. An operator can therefore distinguish intentional placement exclusion from offline storage.

Empty followers can copy a shorter local tail

The release also implements KIP-1023. The new follower.fetch.last.tiered.offset.enable setting defaults to false. When enabled, empty followers can bootstrap using the offset after the portion already uploaded to remote storage.

Previously, tiered-storage follower bootstrapping used the leader's local log start offset. If the leader retained substantial data locally that was already available remotely, a new follower could copy that data again. Starting at the remote boundary reduces the local data that the follower needs to replicate before catching up.

The follower still needs auxiliary state. KIP-1023 describes rebuilding leader-epoch information and producer snapshot state for the historical portion. Skipping the local copying of remote-backed segments does not eliminate the state needed to interpret the log correctly.

The proposal introduces an earliest-pending-upload offset query. That boundary belongs to the leader because it knows which segments have reached remote storage. Merely adding one to a previously fetched offset would also require obtaining the appropriate leader epoch and would make future handling of offset gaps harder.

Faster catch-up changes local availability

A newly bootstrapped follower can have fewer local segments. If it becomes leader, reading older data may require remote fetches. KIP-1023 names this tradeoff explicitly and leaves additional leadership-selection mechanisms outside its scope.

The same concern applies when consumers fetch from a nearby replica. A follower with a small local history may need remote reads to serve those requests. Faster replication catch-up and fast access to historical records are separate properties, so the new option should not be described as improving every read path.

These two storage changes address different phases of maintenance. Cordoning stops new placement on a destination being retired. Tiered-offset bootstrapping can reduce what an empty replacement replica copies locally. Together they make capacity changes more explicit, while leaving replica movement and the performance of historical reads as distinct operational concerns.