Kafka tiered storage separates retained history from local disks

Keeping more Kafka history normally means retaining more log data on broker disks. That couples the period available for replay to the storage attached to the machines serving current traffic. Kafka 3.6 introduces an early-access implementation of KIP-405 that separates a local tier from remote segment storage. The October 10 release announcement identifies tiered storage as suitable for non-production testing. Its significance is the new relationship between a broker and the history that broker can serve.

The 3.6.0 operations documentation describes the workload behind the design. Consumers commonly read near the end of a log, where the operating system's page cache can serve recent data. Older records tend to be needed for backfills or recovery. A local disk policy that treats both workloads identically makes the less frequent historical read compete for the same storage allocation as current traffic.

I value Kafka's replay model because a consumer can revisit retained history without changing another consumer's position. Tiered storage interests me because it changes the physical cost of making that history available. The useful design question is how far retention can extend independently of the broker's local disk allocation, while preserving the metadata needed to find and interpret older records.

Completed segments form the remote unit

The remote tier stores completed log segments in an external storage system. The operations guide names systems such as HDFS and S3 as possible destinations. An active segment still belongs to the ordinary local log. After it rolls, the completed segment can participate in the remote lifecycle. Remote storage is therefore part of Kafka's segment management, rather than an application consumer periodically exporting selected records into unrelated files.

That choice preserves a connection between stored data and the partition log it came from. A historical read needs to identify the relevant segment and the position within it. Copying only record payloads would discard information needed to serve that read through Kafka. The implementation's plugin contract consequently includes the segment's auxiliary files as well as its data.

The RemoteStorageManager interface in 3.6.0 describes the copy operation in detail. Its inputs include offset and timestamp indexes, the transaction index, leader-epoch information, and a producer snapshot. Fetch methods expose segment data starting at a byte position, optionally ending at a requested position. A separate method fetches an index of the requested type. These operations describe a storage adapter for Kafka's log representation.

Indexes matter because an offset is a logical position in a partition, while a storage system returns bytes. The broker needs the information that connects those representations. Transaction and leader-epoch information add further context to the segment. A remote plugin must preserve the complete material requested by the contract so that older data remains usable through the broker's read path.

Kafka does not bundle a production storage implementation behind this interface. The operations guide explicitly says that an out-of-the-box RemoteStorageManager implementation is absent. Enabling a broker property alone therefore does not create a working object-store connection. The operator must supply an implementation and configure its class and classpath. Storage credentials and destination-specific behavior belong to the chosen implementation's configuration.

Segment metadata has its own consistency requirement

Remote bytes are only part of the design. RemoteLogMetadataManager maintains metadata about remote log segments with strongly consistent semantics. Kafka supplies a default implementation that stores this metadata in an internal topic. The operations guide allows an alternative implementation through separate class and classpath settings. This is a second plugin responsibility, distinct from copying segment objects.

The storage interface documents why these responsibilities are separate. Kafka records segment metadata before and after copy or delete operations, with the corresponding lifecycle state. A reader needs to distinguish an operation that has started from one that completed. The presence of a partially written object cannot by itself establish that a segment is available for reads. The metadata manager provides the state used to interpret those operations.

Copy and delete also need defined retry behavior. The interface expects these operations to be idempotent. Deleting resources that a previous attempt already removed should succeed, and repeated copying must handle existing content according to the contract. Those requirements let the broker retry work without requiring the storage adapter to infer that every repeated request is a new logical segment.

Each remote segment has an identifier in its metadata. The contract describes that identifier as unique even when topic, partition and offsets coincide. That qualification prevents a plugin from treating a familiar offset range as a complete identity for remote content. The associated lifecycle metadata remains necessary to determine which stored segment an operation concerns.

This division adds an operational dependency that a local-only log does not have. A remote object and the metadata describing it must remain usable together. A successful upload is evidence about one operation, while a successful historical read requires the metadata lookup and the relevant index reads to work as well. This follows directly from the two interfaces and their different responsibilities.

Local retention becomes a separate policy

Tiered storage is disabled by default at the broker. The guide uses remote.log.storage.system.enable to enable the subsystem. It also requires the storage-manager implementation. When the default metadata manager is used, remote.log.metadata.manager.listener.name identifies the broker listener used by its internal clients. These settings establish the broker-side machinery before any topic chooses to use it.

A topic then opts in through remote.storage.enable. Local retention settings determine how much history remains on local disks, separately from the topic's overall retention. The guide introduces local.retention.ms and local.retention.bytes alongside the existing time and byte retention settings. If local values are not supplied, the corresponding overall retention values apply.

For example, the following topic configuration expresses a one-hour local time window and a seven-day overall time window. It assumes brokers already have a working remote-storage implementation and the required metadata configuration. The numbers are illustrative policy choices, not performance recommendations.

remote.storage.enable=true
local.retention.ms=3600000
retention.ms=604800000

The distinction prevents local deletion from necessarily ending a record's useful life in Kafka. A completed segment can leave the local tier while remaining part of the remotely retained history. Overall retention still determines the intended lifetime of the log. Remote storage does not turn a finite retention policy into a promise that every record will remain available indefinitely.

I see that separation as a useful extension of Kafka's existing model. Replay has always depended on the retention window. Giving the local tier its own budget makes the placement of that retained data an explicit decision. The consumer's reason for revisiting an offset does not need to dictate that every retained segment occupy a broker disk for its entire lifetime.

Early access limits the deployment choices

The release's limitations are specific enough to shape an experiment. The tagged operations guide excludes clusters with multiple log directories and excludes compacted topics. It also says tiered storage cannot be disabled at topic level. Disabling the broker subsystem requires deleting topics that have tiered storage enabled. These constraints make the initial choice of test topics part of the experiment's lifecycle.

The compacted-topic restriction matters for interpretation of the feature. Kafka supports workloads that retain a log for historical replay and workloads that use compaction to retain records according to their keys. This early-access implementation does not cover both cases. A result obtained with an ordinary retained topic should not be generalized to a compacted state topic that the documented feature does not support.

Historical reads also need separate visibility from current traffic. The operations documentation supplies remote fetch and copy rates, including error rates, plus metrics for the remote reader task queue and thread-pool idleness. These measurements distinguish a problem retrieving remote history from a problem copying completed local segments. A broker can face different pressure in each direction because those operations do different work.

The feature does not create ordering across partitions. Its design moves completed segments and retains the metadata that lets Kafka retrieve them. Partition choice remains the application's ordering decision. Storage placement changes which systems a read depends on, so a backfill through remote history has a different dependency path from a consumer reading recently appended local data.

Kafka 3.6 provides enough of that path to test the design with a supplied storage plugin. A meaningful experiment can roll completed segments out of a short local window and then read offsets that require the remote tier. That exercises the indexes and lifecycle metadata together with the segment bytes, which is the behavior required for retained remote history to remain accessible through Kafka.