Kafka 4.2 makes share groups ready for independent work items

Kafka's partition model couples consumer-group parallelism to partition count. A workload made of independent jobs can need more concurrent workers without needing more independent ordering domains. KIP-932 identifies that mismatch as a reason users over-partition topics and introduces share groups to address it.

The Kafka 4.2.0 announcement on February 17 declares Queues for Kafka production-ready. This is cooperative consumption of ordinary Kafka topics, with record-level acknowledgements and delivery counts. Multiple workers can consume records from the same partition. The broker tracks which records they have acquired.

I value Kafka's model of durable history because consumers can revisit retained records independently. Partition keys remain central when the application needs ordered processing. Share groups add a useful alternative for independent work items, but the choice needs to follow the application's ordering requirements. More workers sharing a partition changes how its records can finish processing.

Assignment no longer grants exclusive partition ownership

In a consumer group, partition assignment separates the work between consumers. A share group can assign the same partition to several members. The assignment tells a worker where to fetch, while record acquisition determines which particular records it can process. The number of workers can therefore exceed the number of partitions.

KIP-932 describes a share-partition as a topic-partition viewed by a particular share group. If two share groups subscribe to the same topic, each has its own state for that topic's partitions. One group's progress does not consume the other group's subscription. The groups use the same underlying log while maintaining independent delivery state.

The distinction also explains why this feature does not require a new kind of queue topic. The proposal implements queue-like consumption over regular Kafka topics. Topic retention still determines how long records remain in the log. Acknowledgement updates a share group's processing state, so it must not be confused with removing the underlying record for every subscriber.

That is consistent with the part of Kafka I find most useful. A stream can feed different systems without forcing them to share one consumption position. The additional delivery model gives one of those systems a way to distribute independent work more finely.

Acquisition locks separate delivery from completion

When a worker fetches an available record, the broker acquires it for that worker under a time-limited lock. Another worker cannot acquire the same record while the lock remains valid. The consumer then reports whether processing succeeded, needs another attempt, or cannot succeed.

KIP-932 describes successful acknowledgement, release for redelivery, and rejection without another delivery attempt. If the worker does nothing, the acquisition lock expires and the record becomes available again. The expiration allows progress when a consumer fails after fetching records.

This introduces a familiar application concern. A worker can perform an external operation and then lose its chance to acknowledge success. Another delivery can follow. The external operation therefore needs an application-level duplicate-handling policy if repeating it would be harmful. Record acquisition does not establish an atomic transaction with an arbitrary external system.

The broker also counts delivery attempts. A record's delivery history is now explicit state that can support handling work that repeatedly fails. Rejecting such a record is a substantive processing decision, because it removes that record from further delivery within the share group. Applications should distinguish a permanent failure from a temporary dependency outage.

RENEW supports work that outlasts its initial lock

Kafka 4.2 adds the RENEW acknowledgement for longer processing. The 4.2.0 acknowledgement enum defines ACCEPT, RELEASE, REJECT, and RENEW. Its RENEW description says the record is still being processed and the acquisition lock should be extended.

The following fragment illustrates the acknowledgement choices for a record already obtained by a share consumer. It is an API illustration, with consumer and record supplied by the application's consumption loop:

consumer.acknowledge(record, AcknowledgeType.RENEW);
// Continue processing the acquired record.
consumer.acknowledge(record, AcknowledgeType.ACCEPT);

Renewal and acceptance express different facts. Renewal says the worker still needs its claim on the record. Acceptance says processing has succeeded. An implementation that renews indefinitely without making progress would keep work unavailable to other workers, so renewal belongs to the application's processing policy.

The release announcement associates this addition with KIP-1222 and explicit acknowledgement mode. That scope matters when adopting the API. An application needs a consumption loop that intentionally manages acknowledgements, including what happens when the worker cannot finish.

The useful consequence is finer control over variable-duration tasks. Increasing a global lock duration for every record makes failure recovery slower for short jobs as well. Renewal lets long work explain why it still holds a record, while expiration remains the recovery mechanism when that explanation stops.

Batching and record limits expose an efficiency tradeoff

Record-level delivery does not mean every request must operate on one record. KIP-932 designs for batches because Kafka's log and fetch path already work with record batches. In the common case, a batch of available records can be acquired and acknowledged together. Partial failures create gaps that subsequent delivery must handle.

Kafka 4.2 adds KIP-1206's share acquire modes. The release describes batch_optimized as a soft limit and record_limit as strict enforcement of the requested record quantity. Those modes expose a decision about batching efficiency and how tightly the consumer needs to bound each fetch.

A worker with a limited processing budget may need a strict record count. A throughput-oriented worker may prefer complete batches. The correct choice follows the worker's ability to hold and process acquired records. It should not be inferred solely from the number of consumer instances.

The broker limits acquired records per share-partition as well. That limit prevents one partition's in-flight state from growing without bound through continued acquisition. When the limit is reached, additional fetches cannot simply keep granting records. Acknowledgements and lock expiration release capacity for progress.

Ordering within a returned batch also needs a precise interpretation. KIP-932 guarantees increasing offsets for records returned in a batch from one share-partition. It provides no offset-order guarantee between batches. That distinction rules out treating parallel workers' completion order as partition order.

The broker owns delivery state

KIP-932 separates group membership from record-state persistence. The group coordinator manages share-group members and their assignments. A share-partition leader, colocated with the topic-partition leader, manages the in-flight records. A share coordinator persists share-group state on an internal topic.

This arrangement explains why a share group is more than a client library that dispatches a fetched batch to local threads. The cluster must preserve delivery state across failures and coordinate claims made by independent consumers. That introduces broker work associated with acquisition, acknowledgement, and persistence.

Kafka 4.2 adds adaptive batching for group and share coordinators under KIP-1224. The announcement describes workload-dependent adjustment of append linger times, removing a fixed five-millisecond latency floor. KIP-1226 adds share partition lag persistence and retrieval, giving operators visibility into consumption progress.

Those additions address the operational consequences of making the broker responsible for more consumption state. Lag remains useful, but the delivery model also needs an explanation of records held in flight and records becoming available again. An application using shares has more to describe than a single committed offset.

Streams also moves coordination toward the broker

The same release makes the Streams server-side rebalance protocol generally available with a limited feature set. KIP-1071 moves task assignment and topology metadata into broker-side coordination. The announcement identifies dedicated RPCs and administrative tools as part of that design.

Share groups and Streams groups solve different problems. Shares distribute independent records among workers. Streams coordination manages tasks associated with a processing topology. Both changes make the broker responsible for coordination that cannot be understood solely by looking at an application's poll loop.

The 4.2 announcement also adds dead letter queue support to Streams exception handlers through KIP-1034. That gives a processing application another explicit way to describe records it cannot process. It does not change a share consumer's acknowledgement semantics, and these APIs should remain separate in an application design.

For independent jobs, share groups remove a specific reason to create extra partitions solely for worker parallelism. For a stateful ordered application, partitioning still needs to reflect the state and ordering it requires. I see the new delivery model as a useful extension of Kafka's retained log, with the acknowledgement and renewal behavior forming part of the application's correctness contract.