Kafka tracks a proposal for sharing records within a partition

Kafka's partition assignment model couples the number of active consumers in a group to the number of partitions they can own. That is useful for ordered consumption, but it makes partition count part of worker capacity planning. Independent work items can need more processing concurrency without needing more independently ordered logs.

KAFKA-16092, created on January 8, tracks development of KIP-932, Queues for Kafka. The proposal as revised on January 8 remains under discussion. This is an implementation-tracking event for a proposed design, not an announcement that a released Kafka version already supplies its APIs.

Kafka's durable log model is the part I find most useful. Consumption can leave history available for other readers and later replay. The proposal interests me because it tries to retain the topic as the stored history while changing how a group distributes records to workers.

Sharing records changes the concurrency unit

KIP-932 proposes a new kind of group called a share group. Its members cooperatively consume records, and their number can exceed the topic's partition count. The motivation explicitly identifies over-partitioning for peak consumer concurrency as a problem the design aims to address.

That makes the proposed capability different from adding another conventional consumer group. Separate share groups independently consume the topic, while members within one share group cooperate on its work. The topic continues to be a Kafka topic, rather than a new storage object called a queue.

The design also distinguishes permission to fetch from exclusive ownership of a partition. Its simple server-side assignor can direct every member toward all partitions matching its subscriptions. Several members can then fetch different available records from the same partition.

The coordinator still needs to manage membership and provide assignments, but those assignments do not establish the same exclusive separation as ordinary consumer-group ownership. A change in which partitions a member should fetch therefore has different coordination requirements. The draft describes avoiding the synchronization barrier associated with transferring exclusive work ownership.

Acquisition is a record state

When a member fetches records, the proposed broker acquires them for that consumer under a time-limited lock. An acquired record is unavailable to another member until the relevant state changes. This provides the exclusion needed for distributing individual work items without reserving the whole partition for one worker.

The consumer can acknowledge successful processing, release the record for another attempt, or reject it as unprocessable. Doing nothing eventually releases the acquisition when its time limit expires. The draft gives a default lock duration of 30 seconds, with configuration controlling the duration.

These are proposed semantics, so the values and interfaces remain subject to discussion. They nevertheless make the design's recovery mechanism concrete. A worker's failure does not require retaining its acquired records indefinitely, since lock expiry makes unfinished work available again.

The broker also limits how many records can be in flight for a share-partition. A share-partition is the group's view of a topic partition, with state separate from another group's view of that same partition. Bounding this state prevents the proposed worker model from implying an unlimited number of simultaneously acquired records.

Redelivery weakens the consumption order

The draft explicitly discusses out-of-order delivery between batches. If one worker acquires earlier records and fails while another completes later records, the earlier records can return on a subsequent fetch. Offsets observed by a consumer need not increase monotonically across those fetches.

That is a meaningful tradeoff for Kafka users. The storage still has partition offsets, but concurrent processing and redelivery do not preserve one worker's sequential progress through the partition. Applications choosing this model would need work items whose processing tolerates those semantics.

The draft retains increasing offset order within a returned batch for a particular share-partition. That narrower guarantee is compatible with a later batch containing redelivered earlier offsets. Keeping the batch guarantee separate from cross-batch behavior avoids describing the proposal as either completely unordered or equivalent to existing sequential consumption.

My interest in Kafka comes with attention to the partition key and its ordering consequences. A share group would change which consumption guarantees the application can infer from that key. More workers on one partition are useful for independent tasks, but they do not provide the same processing order as exclusive partition consumption.

Delivery counts add another persisted responsibility

The proposal gives each record a delivery count and defines available, acquired, acknowledged and archived states. Repeated unsuccessful delivery can move a record into the archived state once the configured attempt limit is reached. The January draft gives five attempts as its default limit.

It also says those count updates are not exactly-once operations. The count helps contain repeatedly unprocessable records, but it is not a precise audit of every delivery in all failure cases. That distinction is part of the proposal itself, rather than an inference that every retry must produce a perfectly durable counter increment.

The share-partition leader would manage and persist in-flight state alongside its view of the partition. This adds broker-side responsibilities beyond retaining the records in the log. Distributing work per record requires remembering which records are currently eligible and which have completed for that group.

Retention still belongs to the topic

The January draft bounds a share-partition's start offset by the topic partition's log start offset. Retention can therefore remove records that have not yet completed through a share group. Size-based retention is particularly explicit about that possibility in the proposal.

This preserves an important consequence of using ordinary Kafka topics. A share group does not independently retain every pending work item forever. Several share groups consume the same stored history, and the topic's retention policy applies to them all.

Opening the tracker gives this design an implementation reference while discussion continues. The substantive proposal is a new consumption model with per-record state, independent worker scaling and redelivery semantics. Its relationship to topic retention and partition ordering is what determines which queue-like workloads it could serve.