Kafka 4.1 share groups distribute records beyond partition ownership

Assigning a partition to one consumer makes the partition count a natural limit on active consumers in a group. Kafka share groups relax that ownership model. Several consumers can share work from a partition, with individual records acquired temporarily and acknowledged separately.

The Kafka 4.1 download entry dates the release to September 2. The September 4 announcement advances Queues for Kafka, KIP-932, to preview. It explicitly says the feature is not ready for production, while inviting evaluation.

I consider the partition key the main design decision in many Kafka applications because it establishes where ordering exists. Share groups are interesting because they target work distribution with a different consumer relationship to that ordered log. They should be evaluated through that difference, without assuming they preserve every processing property of exclusive partition ownership.

Record acquisition replaces exclusive ownership

The design documentation shipped with 4.1.0 describes share groups alongside traditional consumer groups. Partitions can be assigned to multiple consumers, and the number of consumers can exceed the topic's partition count. The unit of acquired work becomes the record.

When a consumer fetches an available record, the broker acquires it for that consumer using a time-limited lock. While acquired, the record is unavailable to other consumers. That temporary exclusivity applies to the record rather than requiring one consumer to own every record in the partition.

This changes how parallelism relates to topic layout. Adding workers no longer necessarily requires adding partitions merely to give each worker exclusive ownership of one. The useful limit now also depends on available records and the broker's acquisition controls.

The topic remains a Kafka topic. Share groups consume independently of other share groups accessing it. The feature therefore changes consumption coordination without introducing a separate queue storage object that replaces the log.

That distinction preserves a useful architectural choice. One topic can remain the shared source of events while different groups maintain independent consumption. Choosing a share group concerns how one of those groups distributes its work.

Acknowledgement describes the record's outcome

The versioned design gives a default acquisition duration of 30 seconds, configurable through share.record.lock.duration.ms. If the consumer does nothing, expiry releases the record for another delivery attempt. Failure recovery can therefore progress without waiting indefinitely for the original consumer.

The consumer can acknowledge successful processing, release the record, or reject it as unprocessable. Rejection prevents further delivery attempts for that record. These are distinct outcomes and should remain distinct in an application's processing logic.

A temporary inability to process a record is not necessarily the same as an unprocessable record. Releasing permits another attempt. Rejecting states that further attempts should stop. The broker tracks delivery attempts, which supports handling records that repeatedly fail.

The system is optimized for batches even though acknowledgements concern individual records. Per-record semantics do not imply that every network operation must carry only one record. The implementation can retain efficient transfers while exposing a finer-grained processing outcome.

This is a substantive departure from treating one committed consumer offset as the complete description of a group's progress through a partition. A record can have an acquisition state and an acknowledgement outcome independently of another record being processed by a different worker.

For an application with varying processing times, that distinction can matter. One slow record need not require assigning every later unit of work in the partition to the same worker. The design targets that kind of sharing, while the application must still decide whether independent processing fits its semantics.

Time limits provide recovery without proving side effects

An acquisition timeout lets another consumer receive work after the original lock expires. That helps the broker avoid permanent blockage when a consumer disappears. It does not, by itself, establish whether an external side effect already happened before the failure.

The record can become available again because the broker lacks a successful acknowledgement. The original worker may have failed before processing, during processing, or after completing an external action. Those are different application states even though the broker sees the same missing acknowledgement.

The useful conclusion is that application-level effects still need a deliberate duplicate-handling policy. A timed lock coordinates delivery. It does not make every operation performed against another system part of one atomic transaction with that lock.

The broker also limits how many records consumers can acquire for each topic-partition within a share group. The versioned documentation names group.share.partition.max.record.locks as the control. When the limit is reached, fetching temporarily provides no additional records until acquisitions decrease.

That limit gives the system a way to bound outstanding work. The lock timeout then provides a route for abandoned acquisitions to become available again. Capacity and failure recovery therefore interact through explicit broker state.

My interest in this model is practical. A retained stream can support both history-oriented processing and independent work items, but those uses have different coordination needs. Share groups make the latter relationship explicit instead of forcing every application to equate a worker with a partition.

Preview remains an explicit feature decision

The 4.1.0 upgrade guide says enabling share groups requires upgrading the share.version feature to one with the feature tool. Installing the broker release alone is not the complete activation step.

The guide recommends share groups for records processed individually rather than as part of an ordered stream. That is a useful semantic boundary. The underlying partition log still has order, but distributing records among independent consumers does not establish ordered completion of their processing.

The preview label also remains specific. Kafka's release announcement invites testing and evaluation while withholding a production-readiness claim. It is inaccurate to describe the feature as generally available simply because it appears in a generally available Kafka release.

This distinction is especially important for a system that already uses the traditional consumer model successfully. The new group type supplies another consumption contract. Its presence does not require changing an application whose correctness depends on its current partition ownership and processing sequence.

Streams begins a separate coordinator transition

Kafka 4.1 also introduces KIP-1071, the Streams Rebalance Protocol, in early access. The announcement says it builds on KIP-848 and makes task assignment a first-class part of the protocol. More logic moves to the coordinator, including task assignments and internal topic creation.

That concerns a different unit of work from share-group record acquisition. A Streams application has tasks and internal resources associated with its topology. Moving their coordination into the broker is a change to how that application manages its execution.

The release explicitly says this protocol is not ready for production. It should not inherit the preview label from Queues for Kafka or the GA status of the underlying consumer protocol. Related designs can have different implementation maturity.

Streams also gains ensure.explicit.internal.resource.naming. The announcement describes enforcing explicit names for internal resources so topic names are predictable and topology changes can preserve existing topics. Naming becomes an application compatibility concern when internal topics retain state across deployments.

The configuration addresses that concern directly. Generated names can depend on topology structure. Requiring explicit names makes the identity of a retained internal resource visible in the application's definition.

Other changes make client behavior more explicit

The release adds deadlock protection when KafkaProducer.flush() is invoked from a send callback. Kafka now raises an exception for that call instead of allowing a possible deadlock. This is a concrete change from indefinite blocking to an identifiable unsupported operation.

Connect also supports installing and running multiple versions of connector plugins. The announcement includes connectors, converters, transformations, and predicates in that capability. It gives a worker environment a way to stage plugin version changes without requiring a separate Connect cluster for every version.

These changes fit the release's broader attention to coordination, but they have their own scope. Producer callback safety concerns one execution thread's progress. Plugin versioning concerns code selection in Connect. Neither changes the readiness statement attached to share groups or the Streams rebalance protocol.

Kafka 4.1's new consumption model is therefore specific. A share group acquires records temporarily, acknowledges their outcomes individually, and can use more consumers than partitions. That supplies a different work-distribution contract over the existing topic log, available in preview for applications whose processing semantics fit it.