Kafka 3.0 changes producer defaults and keeps delivery guarantees explicit
Kafka 3.0 strengthens producer defaults and develops the KRaft preview. Its tagged producer code distinguishes acknowledgement settings from activating the idempotent producer state machine.
A producer can wait for replicated acknowledgement and still need a separate mechanism to avoid duplicates after retries. Kafka 3.0.0, released September 21, advertises stronger delivery guarantees by default. The implementation of the shipped Java client deserves a precise reading because acknowledgement policy and idempotent production are different parts of that guarantee.
The release also adds metadata snapshots to KRaft and deprecates older runtimes and message formats. Those changes affect the project's direction, but they do not turn a producer's delivery policy into a broker-wide switch. The client code packaged in an application still decides how that application's producer is configured.
I value Kafka's durable history when the application can explain what a successful send means. A default is useful only when its actual behavior is understood. The distinction between an acknowledgement and protection against duplicate retries is especially important when records become input to several downstream systems.
Acknowledgement controls the completion condition
The ProducerConfig source at 3.0.0 declares acks with the default all. Its documentation describes three acknowledgement policies. With zero acknowledgements, the producer considers the record sent after adding it to the socket buffer. It cannot establish that the server received it.
With acks=1, the partition leader writes the record locally and responds without waiting for all followers. If that leader fails before followers replicate the record, the acknowledged record can be lost. The setting therefore has a weaker completion condition than waiting for the in-sync replicas.
With acks=all, the leader waits for the full in-sync replica set to acknowledge. The configuration documentation describes this as the strongest available acknowledgement guarantee, provided at least one in-sync replica remains alive. The -1 form is equivalent, and the parser normalizes the textual all value to that numeric representation.
These policies concern when a produce request is considered complete. They do not independently determine what happens when a client retries a request whose result was uncertain. Receiving no response is not proof that the broker wrote nothing. That uncertainty is why retry handling needs a mechanism beyond choosing a stronger acknowledgement setting.
Retry protection is another part of the producer
The same source documents enable.idempotence as preventing duplicate writes of retried messages. It requires positive retries, acks=all and at most five in-flight requests per connection. The producer can preserve ordering for the permitted in-flight values when idempotence is active.
Without idempotence, the configuration documentation warns about reordering if several requests are in flight and one batch is retried after another succeeds. Waiting for all replicas does not change that ordering scenario by itself. The acknowledgement policy and the producer's retry identity solve related but different problems.
There is a detail in the shipped code that should not be hidden by the release summary. The configuration definition declares enable.idempotence with a default of true. However, the idempotenceEnabled() method checks whether idempotence was explicitly supplied in the original properties, then reads the value in that case. It also enables the path when a transactional ID is configured.
The KafkaProducer implementation at the same tag calls that method when constructing its transaction manager. This means the declared default and the predicate used to activate the state machine need to be distinguished. For an application requiring idempotence, setting it explicitly avoids relying on an inference from the configuration definition alone.
This configuration excerpt states the intended policy directly:
acks=all
enable.idempotence=true
max.in.flight.requests.per.connection=5
The retry count is left at its declared positive default. These are producer properties, to be combined with the application's connection and serialization configuration. They do not replace topic replication policy or describe a complete application transaction.
Validation protects combinations of settings
The producer rejects an in-flight limit above five when its idempotence predicate is active. Configuration processing also rejects zero retries for that mode. If acknowledgements were explicitly configured to a value other than all, the implementation raises a configuration exception instead of silently accepting an incompatible combination.
That is useful because the values describe one coordinated policy. A wrapper that exposes a retries option while hiding the idempotence setting can still create a conflicting configuration. The source checks the combined state rather than treating each valid individual value as sufficient.
Transactions introduce another condition. A transactional ID enables the idempotent path unless the caller explicitly disables it, which produces a configuration exception. KafkaProducer constructs the transaction manager using the transactional ID when that path is active. It then distinguishes whether the resulting producer is transactional or merely idempotent.
Retry count also has a time boundary. The configuration documentation says delivery can fail before the retry count is exhausted when delivery.timeout.ms expires. It recommends controlling retry behavior through that delivery deadline instead of reducing the retry count. The number of permitted attempts and the total time the application is prepared to wait describe different constraints on a send.
Batching introduces another interval. linger.ms permits waiting for more records before sending a batch, while reaching the configured batch size sends it without waiting for the remainder of that interval. The source describes this as a way to reduce request count under moderate load at the cost of additional waiting. Neither batching nor its delay changes what idempotence means, but both affect when work reaches the request path whose acknowledgement and retry policies the application selected.
The public methods also preserve that separation. Transaction operations require appropriate producer state, while ordinary sends can use idempotence without the application opening a transaction. A delivery guarantee for retried sends is therefore not a promise that arbitrary application effects are atomic together.
I prefer an explicit policy here because it can be reviewed alongside the code that handles send results. A producer property is part of the application's behavior, even when a library offers a default. Naming that requirement also makes it easier to spot a wrapper or deployment setting that changes the intended combination.
The consumer and administration contracts move too
The 3.0 upgrade notes increase the default consumer session timeout from ten seconds to forty-five seconds. They also change the default assignment strategy list to include both RangeAssignor and CooperativeStickyAssignor. Range remains the default selected behavior, while removing it from the list can support a transition to cooperative assignment.
That is another example of a default list needing interpretation. Having a strategy present does not establish that the group has already switched to it. The upgrade note explains which strategy remains selected and how the configuration supports a later change.
Administrative tooling removes direct ZooKeeper options from topic and partition reassignment commands, directing users toward --bootstrap-server. Connect also removes its deprecated internal converter settings and fixes the internal representation to JSON without schemas. These are concrete compatibility changes for surrounding automation, separate from the producer's send policy.
KRaft develops its metadata recovery model
The release summary identifies metadata snapshots among KRaft improvements. The 3.0 KRaft guide still labels the mode a preview for testing. It does not support upgrading an existing ZooKeeper cluster into KRaft, and the upgrade notes exclude upgrading from the 2.8 early-access cluster.
Controller membership remains an explicitly selected quorum. A majority must stay alive for availability. Combined broker and controller processes are simpler for small experiments but share resource failures, including a broker-triggered out-of-memory condition. Those responsibilities remain even as the metadata mode gains more complete recovery machinery.
The guide also retains explicit storage formatting with a shared cluster identifier. It explains that a missing mount can make a directory look empty, and automatic initialization could otherwise hide lost metadata history. That design choice concerns preserving cluster state before a node starts serving, while snapshots address how retained metadata can be recovered efficiently.
Kafka 3.0 changes several defaults and compatibility boundaries, with different effects in each component. For producers, the acknowledgement default is visible in configuration, while the tagged activation path warrants explicit idempotence settings when that behavior is required. The application's definition of a completed send still needs to match the actual producer state machine it creates.