Kafka Streams 4.3.1 closes leaked RocksDB native resources

A small Java options object can own a substantial native allocation. KAFKA-20616 reports that Kafka Streams created options for a RocksDB offsets column family and then lost the reference needed to close them. The native construction also created a default table factory and an eight-megabyte LRU cache. Repeating the lifecycle could therefore grow process memory even though the Java code appeared to discard its temporary object.

The Kafka 4.3.1 announcement on June 25 highlights the fix as the release's most notable correction. The issue marks 4.3.0 as affected and 4.3.1 as fixed. It also describes a second leak in exception handling during store closure.

I value Kafka's ability to retain history and let consumers rebuild their state. That model becomes operationally expensive if repeated state-store lifecycles retain native resources. This fix matters because processing correctness includes releasing the resources associated with a store, including when its final state write fails.

The offsets column family had its own options

The implementation patch explains why the options existed separately. The offsets column family stores small metadata and does not need the data column family's larger buffers and compaction settings. Reusing heavyweight data options could add write amplification and pressure during restore.

Separating the options was therefore a reasonable allocation decision, but it introduced an ownership obligation. The old helper returned a new ColumnFamilyOptions object that was passed into a descriptor. It did not retain that object in a store field, so the store's close path could not release it.

The issue reports that the leak multiplies per segment and per task. Windowed and segmented stores can amplify it because they create more underlying storage instances. A small amount of leaked ownership at one construction site can therefore become substantial in a topology with many such instances.

The fix retains the options on an offsetsCfOptions field. Store closure explicitly closes that object and clears the reference. Cleanup after initialization failure also closes the options if they were created. Ownership now follows the store through both ordinary operation and partial setup.

The patch additionally configures the offsets column family without a block cache. That matches the small metadata workload described in the source. Disabling an unnecessary cache reduces the allocation, while retaining and closing the options addresses the underlying lifetime requirement.

A failed close-time write skipped handle release

The second leak concerns a sequence inside AbstractColumnFamilyAccessor.close. The accessor writes a closed-state marker into the offsets column family, then closes its native handle. If the write throws, ordinary sequential control flow never reaches the close call.

KAFKA-20616 associates that failure with unclean shutdown and exactly-once-v2 failure cascades. The existing code already acknowledged that RocksDB could be in recovery or a failed state. The missing protection was around the cleanup that still had to occur after the metadata write.

The corrected ordering places handle release in a finally block. In simplified Java control flow, the pattern is:

try {
    persistClosedState();
} finally {
    releaseHandle();
}

Those method names illustrate the control flow, not a new Kafka API. The important property is that failure in the state write cannot bypass handle release. Successful metadata persistence and resource cleanup have separate obligations.

The single-column-family accessor had the same dependency on its superclass completing normally. Its data handle closed after the superclass call. The dual-column-family accessor likewise needed to close both its old and new handles after attempting superclass cleanup. The fix uses nested finally blocks so an earlier failure does not skip later releases.

Suppressed shutdown errors made the leak quiet

The issue notes that RocksDBStore.close catches the resulting RocksDBException. That behavior can keep the exception from becoming an obvious application-level failure while native handles remain allocated. A close operation that returns to its caller therefore did not prove that all native resources had been released.

The regression tests observe ownership directly through isOwningHandle(). They capture the options or column-family handles while the store is open and assert that those objects no longer own native handles after closure. That is a more specific assertion than checking whether close returned without throwing.

The tests also inject a failure into the offsets write. This exercises the exact control-flow branch that previously skipped cleanup. Ordinary open-and-close tests would not reach that branch reliably, so they could pass while leaving the failure-path leak intact.

The patch covers migrating stores as well. Their dual accessor holds an offsets handle plus old and new data-column-family handles. The test must confirm release of every applicable handle when the closed-state write fails, because closing only the first one would leave part of the native allocation problem unresolved.

Heap size does not account for the whole store

The reported allocations live on the RocksDB native side. The issue's title identifies eventual out-of-memory failure, but the mechanism is not simply an ever-growing Java collection. A heap-only explanation would miss the native objects retained by unclosed handles and options.

That distinction changes what constitutes useful diagnostic evidence. Process memory growth alongside store creation and closure is relevant to this issue. The number of segments and tasks also matters because the report describes multiplication across those instances. No universal growth rate follows from the eight-megabyte cache figure alone.

Increasing a memory limit can delay exhaustion without closing the leaked resource. Similarly, reducing the workload may change how often the affected lifecycle occurs without correcting its ownership. The release fix changes the resource lifetime and exception ordering that produced the leak.

For applications on the affected Streams release, the corrected library is the relevant upgrade target. Updating a broker does not replace the Streams implementation packaged into an application. The useful deployment record therefore identifies the application's Streams version alongside the state-store configuration.

The source-level result is specific. Offsets options now have an owner that closes them, and close-time write failures no longer bypass the associated column-family releases. That gives stateful applications a corrected lifecycle for ordinary closure and the failure paths described in the issue.