Kafka 0.11 commits output records and consumer progress together

A consumer that writes an output record and then saves its input position has two results to make consistent. If it crashes between them, its replacement can repeat work whose output already exists. Reversing the order creates the opposite failure, recording progress before producing the corresponding output. Kafka 0.11 adds transactions that can commit those results together when the output is another Kafka topic.

The 28 June release entry marks the availability of 0.11.0.0. Its design documentation explains two new capabilities, an idempotent producer and transactional writes across topic partitions. They address different failure cases, and their scope determines what exactly-once processing means in this release.

I value Kafka's retained log because a consumer can return to an earlier position and replay it. Correct recovery also needs a rule for the output already produced before a failure. This release makes that relationship explicit inside Kafka instead of leaving every topic-to-topic application to reconcile the two results independently.

Producer retries have an identity

A producer can lose the broker's response after the broker has accepted its write. Before this release, resending in that situation can put the same record into the log again. The sender cannot infer from the missing response whether the first write took effect.

The design documentation says the idempotent mechanism assigns the producer an identifier and uses sequence numbers to deduplicate messages. This gives a retried request an identity the broker can recognize. The broker can distinguish another transmission of the same producer operation from a new operation.

The Java producer documentation in the release source enables this through enable.idempotence=true. It requires no new send API. Applications can retain the ordinary producer interface while the client and broker handle retry deduplication.

The same documentation warns against application-level resends. A new send call is not automatically recognized as the same business event merely because its bytes resemble an earlier record. The documented guarantee also applies within one producer session. Idempotence is therefore specific to the producer's protocol identity, not a universal duplicate detector for application events.

Configuration constrains the idempotent producer

In this initial implementation, enabling idempotence defaults retries to the maximum integer value, acknowledgment to all in-sync replicas and the maximum in-flight request count to one. These are version-specific requirements. A description based on a different client version could easily obscure the constraints of the implementation shipping today.

The settings connect retry handling to ordering and acknowledgment. The producer needs to preserve a sequence the broker can interpret and receive a durable result under its configured replication policy. Merely setting one convenient application flag does not make contradictory producer settings meaningful.

The producer documentation also discusses failure after retries, including a message expiring in the buffer before transmission. It recommends shutting down the producer and checking the last produced contents where necessary. This qualification keeps the guarantee tied to the mechanism rather than treating every possible application recovery decision as automatic.

Idempotence closes the retry gap for production. It does not by itself commit several output partitions atomically or connect output to a consumer's position. Those responsibilities require the transactional layer introduced alongside it.

Transactions include several partitions

A transactional producer sets transactional.id, initializes its transaction state and uses explicit begin, commit and abort operations. Setting the identifier also enables idempotence and its dependent configuration. The identifier supports recovery across sessions and must be chosen to identify the producer instance within the application's partitioned arrangement.

Only one transaction can be open per producer. The documentation says all sends between begin and commit belong to that transaction, and a producer configured with a transactional identifier must send within transactions. The API supplies a boundary around a group of outputs instead of treating each send as an unrelated commitment.

The documentation recommends durable topic configuration, including replication factor of at least three and a minimum of two in-sync replicas. Transactions coordinate visibility and outcomes, but the underlying records still need a replication policy appropriate to the required durability. Atomicity does not eliminate storage failure considerations.

Errors also have distinct meanings. The producer example treats fencing and other unrecoverable failures as reasons to close the producer, while other Kafka exceptions can lead to aborting the transaction. A producer cannot continue as though every exception merely asks for another unqualified send.

Committed isolation controls what consumers observe

Transactional records need a corresponding read policy. The design document states that the default read_uncommitted isolation exposes records even from aborted transactions. A consumer that expects transactionally completed output must instead select read_committed. That selection is a concrete consumer property:

isolation.level=read_committed

The setting does not mean the consumer sees only transactional records. The documentation says it returns committed transactional messages and messages outside transactions. Its purpose is to exclude output whose transaction did not commit, while preserving ordinary nontransactional traffic.

This distinction matters to a pipeline mixing producers. One producer's use of transactions does not cause every downstream consumer to acquire the intended visibility policy automatically. A complete description includes the producing transaction and the consumer isolation through which its results become observable.

Offsets can join the output transaction

Kafka stores consumer positions as records too. The design document explains that a topic-to-topic application can write its offsets in the same transaction as its outputs. Committing then establishes both the produced results and the recovery position associated with them.

If the transaction aborts, committed readers do not observe those transactional output records, and the consumer's committed position stays at the earlier value. A replacement can process the inputs again without exposing both the aborted output and its successful replacement through committed isolation.

This is the mechanism behind exactly-once processing between Kafka topics. The application may execute processing code again after a failure, but the committed result and progress remain coordinated. The useful guarantee concerns visible completed effects within this transaction boundary.

I find that formulation more informative than saying a function can never run twice. Replay is already part of Kafka's recovery model. The transaction gives replay a way to repeat computation while retaining one committed outcome for the coordinated Kafka outputs and offsets.

Kafka Streams applies the mechanism through configuration

The Streams configuration source at this release adds processing.guarantee. Its default is at-least-once, and exactly_once selects the new processing guarantee. Installing the library alone does not change an application's selected mode.

The same source enforces associated producer and consumer behavior for the exactly-once setting. It requires committed reads and idempotent production with one in-flight request per connection. Streams configures the client relationships needed by the processing guarantee instead of expecting users to independently discover every setting. A configuration fragment expresses the selection:

processing.guarantee=exactly_once

The broker version still matters. The Java producer documentation says the transactional APIs require brokers from 0.11 onward, even though the client can communicate with some older brokers for other operations. General client compatibility and support for this particular API are separate questions.

External effects require cooperation from their destination

The design document explicitly separates output to Kafka from output to another system. Updating an external database and recording a Kafka consumer position still need coordination. Kafka's transaction cannot automatically include arbitrary effects in a system that does not participate in it.

The document describes storing the consumer position alongside the output in the destination as one possible approach. The relevant property is that the output and its recovery position succeed together there. That is a different transaction boundary from Kafka's own output topics and offset records.

This limit does not diminish the topic-to-topic mechanism. It defines where the implementation can make its atomic commitment. A pipeline can use Kafka transactions for its internal stages and still need a separate strategy for a final external write.

Kafka 0.11 supplies identities for safe producer retries and transactions for coordinated Kafka results. The consumer's isolation determines whether aborted output remains visible, and Streams can configure these relationships through its processing guarantee. For an application built around replayable history, the substantive change is that output records and the position authorizing recovery can now have one committed outcome inside Kafka.