Kafka 2.7 adds aggregation windows aligned to event timestamps

Two events can be close together in time and still fall into different fixed buckets. That is a problem when the application asks whether they occurred within a particular distance of each other. Making the buckets smaller changes the approximation, but leaves the bucket boundaries part of the answer. Kafka Streams 2.7 adds an aggregation window whose boundaries depend on event timestamps.

Apache's Kafka 2.7.0 release entry dates the release to December 21. Sliding-window aggregation is among its additions, alongside PEM certificate and private-key support. The release also introduces controls for broker connection creation rates and throttling of topic creation, partition creation, and topic deletion. The window addition changes what a Streams application can express directly, so its semantics deserve more attention than the name alone provides.

I treat Kafka's retained history as useful only when the application is precise about its interpretation. Window boundaries are part of that interpretation. A replay that reads every original record can still answer the wrong question if the aggregation groups those records differently from the intended rule.

Windows follow changes in membership

The 2.7 Streams DSL guide distinguishes sliding windows from hopping and tumbling windows. Sliding windows align to record timestamps. They use a fixed maximum time difference between records, with both interval boundaries inclusive. Each unique combination of included records appears in one window snapshot, although an individual record can participate in several snapshots.

That last distinction prevents an easy misunderstanding. A sliding-window count does not partition the input into disjoint groups. The application is asking about overlapping sets of nearby records. Adding the counts across those sets would count some records more than once, which makes the result unsuitable as an overall event total.

The release's SlidingWindows class documentation gives a concrete example. For one key, records arrive at timestamps 8,000, 9,200, and 12,400 milliseconds. With a maximum difference of 5,000 milliseconds, all three fit into the inclusive interval beginning at 7,400 and ending at 12,400. Their earliest and latest timestamps differ by 4,400 milliseconds.

The same example also contains a snapshot beginning at 8,001 and ending at 13,001. It excludes the first record and contains the other two. Another begins at 9,201 and ends at 14,201, containing only the third record. Those starts follow the moments when an earlier record drops out. Membership changes determine which snapshots are needed.

This is a different model from periodically evaluating an interval at a chosen hop. A hopping window's evaluation schedule can affect which combinations it captures. Sliding-window aggregation represents combinations according to the event timestamps and the configured maximum difference. The choice between them should follow the question the application needs to answer.

Grace is a separate time decision

The new API requires an explicit grace period. This fragment uses the factory method present in the 2.7.0 source and defines a five-second maximum timestamp difference with thirty seconds of grace.

import java.time.Duration;
import org.apache.kafka.streams.kstream.SlidingWindows;

SlidingWindows windows = SlidingWindows.withTimeDifferenceAndGrace(
    Duration.ofSeconds(5),
    Duration.ofSeconds(30)
);

These durations have different jobs. The maximum difference determines which records can belong together. Grace determines how long an existing window can accept out-of-order records as stream time advances. Increasing grace does not turn a five-second grouping rule into a thirty-five-second grouping rule.

The class documentation defines closure precisely. A window closes when stream time is greater than its end plus the grace period. That is a stream-time rule, so thirty seconds of grace does not promise a result thirty seconds after a wall-clock timer starts. An application that needs a deadline in elapsed real time must account for that distinction.

I prefer the explicit grace argument here. A grouping rule alone leaves the treatment of delayed records unspecified. Putting both durations in the window definition makes it possible to review the event relationship and the accepted lateness as separate decisions.

Windowed computations can also update their results while new data arrives. The DSL guide distinguishes those updates from a final result emitted after closure through suppression. A downstream system that acts on each update needs different behavior from one that requires a final count. Sliding windows define membership, while the topology still determines how to consume the changing aggregates.

PEM changes credential packaging

Kafka 2.7's security documentation adds PEM as a key and trust store format for brokers and clients. Credentials can be supplied directly through configuration, and PEM also works for file-based stores. Direct configuration uses ssl.keystore.key for the private key and ssl.keystore.certificate.chain for its certificate chain. Trusted certificates go in ssl.truststore.certificates.

This removes a conversion step for deployments whose certificate tooling already produces PEM material. It also changes where sensitive material may reside. Supplying a private key through configuration makes protection of that configuration part of protecting the key. Kafka's documentation recommends encrypting or externalizing production configuration when an unencrypted key is supplied this way.

The password settings require care because their names are similar. PEM does not use ssl.keystore.password or ssl.truststore.password. An encrypted private key instead uses ssl.key.password. The default SSL engine factory supports a limited set of encrypted-key formats, so accepting PEM does not imply accepting every encryption method an external tool can produce.

I see the practical value in using the certificate representation that the surrounding deployment already manages. The change reduces packaging work, while certificate trust and private-key handling retain their existing importance. It does not make the choice of trusted certificate authorities or the contents of a certificate chain an application detail that can be omitted.

The connection and administrative throttles address a separate operational concern in this release. Apache explicitly lists broker-wide and per-listener connection creation limits, along with throttling for the named topic operations. They give operators controls over work that ordinary record traffic limits do not describe. A Kafka deployment's capacity policy can therefore account for connection establishment and topic mutations separately from ongoing production and consumption.