Kafka 2.5 adds co-groups while changing TLS defaults

Combining several event streams into one per-key aggregate should not require pretending that every input has the same value type or update rule. Kafka 2.5 adds a co-group operation to its Streams DSL for that case. The 15 April release summary also lists TLS 1.3 support, incremental consumer-rebalance work and operational metrics. The co-group API is a useful place to examine what the release changes in an application's model of state.

The 2.5.0 release notes identify the DSL work as KAFKA-6049. A co-group combines grouped streams before aggregation. Each input supplies an aggregator, while the output has one aggregate type. That preserves input-specific update logic without requiring the application to construct independent tables and combine their results afterward.

I value Kafka's model of retained history because different consumers can derive different views from it. A co-group gives a Streams application a direct way to define one view updated by several inputs. The key and the aggregate's meaning remain design decisions, because adding an operator does not establish the relationship between the events arriving on different streams.

Each input has its own aggregate update

The released CogroupedKStream interface describes an intermediate form between grouping and aggregation. The first grouped stream creates it with an aggregator. Further grouped streams join through cogroup calls, each supplying an aggregator appropriate to that input's value type. Calling aggregate supplies an initializer and produces a KTable of the current result by key.

The initializer runs once per key immediately before the first input record for that key. Subsequent records apply the aggregator belonging to their input stream to the current aggregate. This makes the shared state explicit. Different input types can update the same output type without making their input records interchangeable.

Consider an illustrative application in which additions and removals update an inventory total. Those are two conceptual inputs with different effects on one aggregate. A co-group can express the separate update functions while retaining a shared key. This is a description of the API's role, not a claim that the operator supplies an inventory data model or determines the correct behavior for duplicated business events.

The distinction is useful because an aggregate is a computation over the records the application processes. A business interpretation needs its own event definitions. The API provides a way to place those definitions in separate functions, so the common output does not require erasing the distinction between inputs.

Grouping still has partition requirements

The interface requires grouped streams in the co-group to have the same number of partitions. Its documentation points to writing through a pre-created topic with the appropriate partition count when that is not already true. The grouping operation therefore does not eliminate the physical requirements for placing corresponding keys together.

That requirement connects the DSL to Kafka's partitioned log. Per-key computation needs the relevant records to reach the corresponding stateful work. A method that accepts several streams cannot infer that differently partitioned topics already satisfy that requirement. The application definition and topic layout have to agree on how the grouped records meet.

Null keys or values are ignored by this aggregation API. The result is stored in a local key-value store, with an internal changelog for recovery. Updates also flow downstream as a KTable changelog, but the interface warns that caching can deduplicate consecutive updates for a key. Downstream observers should not assume that every intermediate aggregate update produces a separately observable output record.

I find that last detail important when using a table as an event-processing result. The table represents the rolling value. The number of visible update records is affected by caching and commit configuration. An application that needs to preserve every input event already has the source streams, whereas the aggregate's changelog has the semantics documented for the table.

TLS support and TLS defaults are separate changes

The release's security headline also needs a precise reading. Kafka supports TLS 1.3, but the 2.5 upgrade notes say only TLS 1.2 is enabled by default. TLS 1.0 and 1.1 are disabled by default because of known vulnerabilities. Their availability through explicit configuration is different from their inclusion in the default protocol set.

The released SSL configuration definitions distinguish ssl.protocol, used to generate the SSLContext, from ssl.enabled.protocols. They set the default context protocol to TLSv1.2 and describe TLSv1.2 and TLSv1.3 as allowed values on recent JVMs. JVM support and Kafka's selected configuration both matter to whether a connection can use the newer protocol.

Consequently, upgrading to 2.5 does not mean every encrypted connection immediately uses TLS 1.3. Support allows an appropriate configuration to select it. The default change removes older protocols from the normal enabled set. These are different operational effects, and reporting only that TLS 1.3 exists would omit the change most likely to affect an older peer's default negotiation.

Consumer progress continues during some rebalances

The upgrade notes also refine cooperative-rebalance behavior. Consumer poll can return records for partitions still owned during a cooperative rebalance. A synchronous commit can throw the nonfatal RebalanceInProgressException, allowing the application to complete the rebalance and retry the commit for still-owned partitions. The notes distinguish that case from a fatal CommitFailedException.

That change preserves a more specific account of ownership during transition. A consumer can retain useful work on partitions it still owns while the group changes assignment. Like the co-group's partition requirement and the explicit TLS configuration, it makes an existing boundary more precise. Kafka 2.5 adds application-level expressiveness while documenting the ownership and configuration conditions under which the surrounding operations remain valid.