Librdkafka 0.9.2 adds LZ4 and an explicit producer flush

A producer can accept a message before the broker has completed the corresponding request. That gap matters during shutdown, when an application may finish its own work while the client still holds queued or in-flight messages. Librdkafka 0.9.2 adds an explicit flush operation alongside LZ4 support and new event interfaces. The release addresses both the representation of data on the connection and the application's relationship to asynchronous client work.

The 8 November release lists LZ4 compression support under KIP-57. It also adds rd_kafka_flush(), queue-length inspection and file-descriptor-based queue triggering for applications with I/O polling loops. These are client capabilities, so they should not be confused with a new Apache Kafka broker release.

I find Kafka's retained log useful because consumers can manage their own positions and replay history. The producer has an earlier responsibility in that arrangement, getting an event into the log under a defined delivery policy. Client buffering and completion handling are part of that responsibility, even when the application's produce call itself is brief.

Compression applies to message sets

The configuration reference at v0.9.2 adds lz4 to the supported compression codecs. The global producer setting supplies a default for topics, while a topic-specific setting can override it. Compression is configured for message sets, so it participates in the client's batching behavior.

The same reference describes a message-count batch limit and a separate maximum message size. Those controls constrain different dimensions of a batch. A count limit cannot describe the total bytes when payload sizes differ, and a byte limit does not establish a fixed number of messages per request.

The release also changes behavior when compressed data exceeds the size of its uncompressed representation. The producer sends the uncompressed message set in that case. That is a concrete reminder that selecting compression does not guarantee that every payload becomes smaller. The implementation can avoid transmitting a larger compressed representation.

Build capabilities remain relevant. The configuration's builtin.features property lets an application query the compiled library's capabilities or require a list of them. A program that relies on LZ4 can check library support instead of assuming that the presence of a configuration name proves that every installed binary has the same build features.

Flush waits for completion within a bound

The versioned API header defines flush as waiting for outstanding produce requests to complete. It recommends the operation before destroying a producer so that queued and in-flight requests can finish. The timeout is an explicit part of the call.

The return value distinguishes completion from expiration of that bound. Flush returns the timeout error when the deadline arrives before all outstanding requests complete. Otherwise it returns the no-error value. An application cannot interpret merely calling flush as evidence that every outstanding request finished.

The function also calls rd_kafka_poll() and can trigger callbacks. Shutdown therefore remains active application execution, rather than a passive pause that prevents delivery callbacks from running. Callback code must still have the state it needs while flush is in progress.

This contract separates waiting from evaluating delivery outcomes. The client has delivery-report callbacks, and the configuration can request reports only for failures. Those reports describe results that application policy must handle. A completed request and a successful end-to-end application operation are not interchangeable statements.

Retry and acknowledgment policy remain separate

The release's configuration reference states that retries can cause reordering. Adding LZ4 or flush does not remove that qualification. A producer must still understand how retry behavior interacts with the ordering it expects within a partition.

Acknowledgment configuration is another independent input. The topic setting describes no acknowledgment, acknowledgment by the leader, and waiting for in-sync replicas according to the configured policy. Flush waits on the work the client has submitted under those settings. It does not silently replace the chosen acknowledgment requirement with a stronger one.

That distinction is useful when comparing applications that both say they flush before exit. The same lifecycle call can surround requests using different acknowledgment policies and local delivery timeouts. The relevant guarantee comes from the combination of those settings and the reported outcomes.

I prefer the explicit lifecycle operation because it gives shutdown a named completion boundary. It does not require inventing a duration that is assumed to be enough for all pending work. The timeout remains visible, and expiration produces a result the application can handle.

Event integration reduces special-purpose polling

The release adds an event interface and queue operations, including a mechanism for queue triggering through an I/O file descriptor. This lets a program connect client work to an existing I/O polling arrangement. The application still has to consume and interpret events, but it gains another way to know when queue work is available.

Consumer fixes in the same release include avoiding outdated group responses and handling empty assignments. These changes concern a different side of the client, where membership and assignment can change while requests are in flight. They show why client correctness includes state transitions as well as payload serialization.

The immediate producer improvement is specific. An application can select LZ4 where the library supports it and wait explicitly for outstanding produce work before destroying the client. The retained log remains the broker-side history, while librdkafka gives the application clearer control over the buffered work that precedes that history.