Kafka 3.8 makes storage failure and compression choices explicit

Storage capacity, storage failure and compression cost are related operational concerns, but they need different controls. Multiple log directories let a broker use independent devices. Directory-aware metadata tells the controller which replicas a failed device affects. Compression settings determine another tradeoff between processing work and the amount of data retained or transferred.

Kafka 3.8.0, announced July 29, changes both directory handling and compression configuration. KRaft JBOD leaves early access, while per-codec levels give producers and brokers more control than a library default. Other capabilities in the same announcement retain their own preview status.

The retained, partitioned log is the part of Kafka I find most useful. Its operational model still demands attention to which replica owns which data and what each write costs. These changes interest me because they expose concrete decisions at those two points without changing the topic into a different storage abstraction.

JBOD needs directory-level failure information

KRaft JBOD first appeared as early access in 3.7. The 3.8 announcement explicitly removes that designation. The underlying design, KIP-858, explains why support involves more than a comma-separated directory setting.

A broker can remain active after losing one data directory. If the controller only knows that the broker continues sending heartbeats, it can retain partition leadership on unusable storage. The metadata system needs an association between each replica and its directory to identify the affected replicas without declaring every directory on the broker dead.

KIP-858 gives directories persisted UUIDs through directory.id in meta.properties. Broker registration reports available directories, and partition metadata records which directory hosts each replica. A directory identity remains meaningful independently of the filesystem path used to mount it.

Failure notifications can then report directory identifiers through heartbeats. The controller can interpret that information using the stored replica placement. This avoids requiring one failure notification to enumerate every partition on a large disk, while preserving the information needed for leadership and in-sync replica decisions.

Replica movement must maintain the mapping as well. Moving data between two directories on one broker changes the failure domain of that replica. A stale metadata assignment could make the controller interpret the next directory failure incorrectly, so recording placement is part of the storage operation.

Tiered storage keeps a separate status

The release also adds tiered-storage support for clusters configured with multiple log directories. Apache nevertheless keeps tiered storage in early access. The maturity of KRaft JBOD should not be transferred automatically to every feature that now works alongside it.

This distinction is useful when describing an installation. Multiple local directories and remote storage integration solve different problems, and their combined availability does not erase their individual release qualifications. A cluster using only JBOD and a cluster also using tiered storage have different enabled capabilities even on the same Kafka version.

The release adds remote.fetch.max.wait.ms under KIP-1018 to configure a maximum wait for delayed remote fetch requests. That setting concerns the remote retrieval path. It does not replace the directory identities or failure reporting that make local multi-directory operation understandable to the controller.

These changes therefore belong to related storage work without being one universal storage mode. A directory failure, an outstanding remote fetch and the amount of compressed data on disk each have their own representation and control. The release's status labels help keep those responsibilities distinct.

Compression level becomes a supported configuration

KIP-390 introduces compression.gzip.level, compression.lz4.level and compression.zstd.level for producer, topic and broker configuration. Snappy has no corresponding option because it does not expose a compression level.

The codec selects the compression implementation. The level selects how that implementation trades processing effort against compressed size. Those are different decisions, so a deployment can retain its codec while choosing another point within its supported levels for a particular producer configuration.

For a producer using gzip, the proposal gives a configuration of this form:

compression.type=gzip
compression.gzip.level=4

The 3.8 producer configuration source includes the separate per-codec settings. It also documents that compression applies to full record batches. Batch formation therefore remains relevant to compression ratio even after the level becomes configurable.

The new controls do not imply that the highest numeric level is always the best operational choice. KIP-390 frames processing time and compressed size as a tradeoff. The value of reducing bytes depends on the cost of producing those bytes under the selected codec and workload.

A topic level does not force same-codec recompression

KIP-390 specifies when the broker recompresses incoming records. If the topic retains producer compression or uses the same codec as the producer, the broker appends the records without recompressing merely because the producer chose another level. A differing level alone does not trigger conversion.

If the topic requires a different codec, the broker recompresses using its configured codec and level. This is an important limit on the meaning of a topic-level setting. It controls the level used when that compression operation occurs, rather than guaranteeing that every stored batch was recompressed to one prescribed level.

That behavior preserves the benefit of accepting already compressed producer batches. Requiring another compression pass solely to normalize levels would add processing work even when the codec already matches. The proposal explicitly avoids that requirement.

The configuration also handles absent and irrelevant values. Without an explicit level, it uses the existing default. With compression disabled or Snappy selected, the per-codec level is ignored. A supplied level outside the supported range produces an error instead of silently selecting an unrelated value.

Codec-specific names avoid an ambiguous shared range

KIP-390 records rejection of one universal compression.level setting. Each codec has its own valid range, so a number meaningful for one can be invalid for another. Separate property names identify which implementation should interpret the integer.

That choice matters for configuration maintenance. An operator changing compression.type can distinguish the setting intended for the new codec from a number chosen under an old codec's range. The settings describe the actual operation instead of making every integer appear interchangeable across implementations.

The proposal includes benchmark results from a defined dataset and environment, but those results are not a release-wide throughput guarantee. The configuration's durable contribution is giving deployments access to the tradeoff. It does not establish that one level improves every producer's latency or that every workload gains the same reduction in stored bytes.

The benchmark description distinguishes a producer test from a linear write test. They exercise different paths, so even results using the same codec and level answer different questions. Producer throughput includes batching and request behavior that a local write measurement cannot establish by itself. Likewise, a reduction in compressed size does not directly specify the latency experienced by a consumer. Keeping those outcomes separate is necessary to interpret the proposal's motivation without converting one measured result into a broader promise about an entire application pipeline.

I prefer that explicit control to treating a library default as an immutable property of Kafka. A default remains useful when the tradeoff does not justify another setting. Where processing cost and retained size matter, the configuration now names the choice and the codec that interprets it.

Consumer coordination remains a preview

The 3.8 announcement promotes KIP-848 to preview with further fixes and an invitation to test. Its assignment calculation moves into the broker's group coordinator, and incremental rebalancing removes the global synchronization barrier described in the older protocol. This is a separate maturity statement from JBOD leaving early access.

The release also adds client rebootstrap support through KIP-899. That lets a client repeat bootstrap when none of its known brokers are available. Re-discovering brokers and calculating group assignments are different recovery responsibilities, even though both affect a consumer's ability to continue useful work.

For operators, the important reading of 3.8 is consequently component-specific. KRaft JBOD gains a stronger release status, tiered storage extends its supported combinations while remaining early access, and compression levels become explicit per-codec controls. The topic still stores partitioned history, while the release gives the surrounding system more precise ways to describe storage availability and the cost of writing that history.