Kafka 3.9 gives tiered storage operational controls

Offloading a log to external storage is only the beginning of operating it. Administrators also need to control copying, bound remote reads and decide what happens to already archived records when tiering stops. Those operations determine whether storage can change without making retained history ambiguous.

Kafka 3.9.0, announced November 6, marks tiered storage production-ready. The release adds per-topic disablement and remote transfer quotas alongside more visibility into remote offsets. Its controller changes also remove one static-membership constraint for newly configured dynamic quorums.

Kafka's retained history is the capability I value most. Separating that history from the broker's local storage can be useful, but the retention and recovery rules still have to remain explicit. The most interesting part of this release is the control it adds around stored data after initial enablement.

Stopping new copies does not require deleting old history

KIP-950 distinguishes two disablement policies. One stops copying new segments while retaining access to already archived data. The other removes remote data from the active log and schedules its deletion.

The distinction is represented through separate topic settings. Keeping remote storage enabled while setting remote.log.copy.disable=true stops further archival without making previously archived segments unreadable. Expiration work continues for those segments under the applicable retention policy.

This creates a useful intermediate operating mode. A topic can stop adding remote history while consumers still need records already stored there. Treating disablement as immediate deletion would unnecessarily couple those two decisions and could make a change in future storage policy remove existing replay capability.

The destructive alternative sets remote.storage.enable=false together with remote.log.delete.on.disable=true. Remote records leave the active log and deletion proceeds asynchronously. This is a different request from merely stopping new copies, even if both are described casually as turning tiering off.

The proposal explicitly limits the 3.9 implementation to KRaft. Its broader design discusses ZooKeeper as well, but those planned paths should not be mistaken for delivered support in this version. The release capability needs to be read with that implementation boundary attached.

KIP-950 also excludes compacted topics and distinguishes topic disablement from the cluster-wide remote.log.storage.system.enable setting. A per-topic lifecycle operation does not remove the broker's remote-storage subsystem for every topic. That distinction lets the operator reason about one topic's archived history without treating the change as shutdown of the entire storage integration.

Retention must preserve a contiguous log

With tiering enabled, the topic has local-retention controls and overall retention controls. The local settings determine the local portion, while the topic-wide settings describe expiration of the retained history. After copying stops, the remaining remote data still belongs to that history.

KIP-950 explains that expiry must begin at the earliest retained offset. When remote data remains after copying is disabled, it expires before later local data. Expiring newer local data while retaining older remote segments could create a gap in what the topic exposes as its active log.

That requirement also affects re-enablement. Stopping and resuming archival should not produce an offset interval that disappears merely because the storage policy changed twice. The proposal's retention ordering preserves continuity across the remote and local portions.

Deletion has another complication because it is asynchronous. Tiering can be re-enabled while older remote objects are still being removed. The design uses a tiered epoch to separate data belonging to the old deletion cycle from newly archived segments, preventing the two lifecycles from becoming confused.

These rules explain why disablement requires more than changing the copy scheduler. The system must preserve a coherent active log while background work expires or deletes objects. A configuration response and physical removal of every object are not necessarily simultaneous events.

Quotas bound work that can otherwise compete with producers

KIP-956 identifies pressure caused by remote copying. Enabling tiering on an existing topic can expose many eligible segments at once. Concurrent upload tasks can consume CPU while trying to archive that backlog, affecting other broker work.

The read side has a corresponding problem when consumers request old records. Many concurrent remote fetches can consume broker resources and put pressure on the remote storage service. A large replay can therefore have consequences outside the consumer that initiated it.

The proposal adds separate byte-rate bounds for copying and fetching. remote.log.manager.copy.max.bytes.per.second controls remote upload work, and remote.log.manager.fetch.max.bytes.per.second controls remote reads. The proposed defaults are effectively unlimited, so support for quotas does not imply that an installation has already selected a restrictive value.

Both bounds are dynamically configurable through the cluster-default broker configuration. The quota manager tracks aggregate activity on a broker. Configuring a common value across brokers does not turn it into one shared cluster-wide byte counter.

This is another place where configuration scope matters. A per-broker rate affects the amount each broker can transfer, while the total across a deployment depends on how many brokers are doing work. The property name and its configuration location need to be interpreted together.

Throughput and concurrency are different controls

KIP-956 rejects using only thread-pool size as a substitute for transfer quotas. The number of workers limits concurrent work, but it does not directly state a byte rate. Different segment sizes and operation durations can produce different throughput at the same concurrency.

The proposed upload quota records segment bytes and consults a rolling rate before allowing further copying. The manager is shared across upload tasks on the broker. This gives operators a rate-oriented control in the same units as the resource transfer they intend to bound.

The design also identifies a cleanup consequence. If upload work waits under a quota in the same execution path as expiration, expired remote segments can remain blocked behind copying. Separating upload and expiration tasks prevents a copy throttle from automatically becoming an expiration throttle.

Read quotas also have fairness limits. A shared remote-read budget can be consumed by one aggressive client and delay others. The proposal discusses using client fetch quotas alongside the broker's remote-fetch limit so one consumer does not consume the entire budget unchecked.

I find that separation useful because a replaying client and the broker's archival backlog are different sources of pressure. They need independent controls, and neither control establishes fairness by itself. The design names both the resource rate and the level at which it is shared.

Visibility extends beyond a successful fetch

The release announcement identifies KIP-1005's offset visibility and KIP-1057's support for examining remote-log metadata through the dump-log tool. The former exposes information about the remotely stored portion, while the latter makes its metadata available for inspection.

Those additions complement quotas and disablement. Rate settings constrain work, but an operator still needs evidence about where data resides and what metadata describes it. A successful recent fetch does not establish the state of an older remote interval or the progress of background storage operations.

The announcement's production-ready status therefore arrives with controls over a fuller storage lifecycle. Enabling archival, preserving reads after stopping copies and inspecting remote metadata are separate operations. Their explicit representation helps make a topic's retained history understandable while its physical storage changes.

Controller membership becomes dynamic within a defined boundary

Kafka 3.9 also introduces dynamic KRaft controller quorums through KIP-853. The release announcement says administrators can add or remove controllers with kafka-metadata-quorum.sh or the associated AdminClient interface. Previously, a static membership configuration had to be supplied across brokers and controllers.

The important limitation is conversion. Kafka 3.9 does not support converting an existing static metadata quorum into a dynamic one. Upgrading an existing cluster to the release does not by itself remove that restriction from its current quorum configuration.

The same announcement identifies 3.9 as the final feature release supporting ZooKeeper mode and as a bridge for migration before a future 4.0 upgrade. ZooKeeper migration must occur in a release that still implements it. The planned removal in 4.0 therefore makes the intermediate release part of the upgrade path.

These metadata changes have a different role from remote log storage. Controller membership determines who maintains cluster metadata, while tiered storage determines where topic data resides and how it is accessed. Kafka 3.9 adds operating controls to both, with explicit boundaries on which transitions the release can perform.