Kafka 1.1 avoids repeating unchanged partition metadata in fetches
Incremental fetch sessions reduce metadata repeated between Kafka brokers. A session identifier and epoch keep the shared state synchronized, while a bounded broker cache limits the memory cost of remembering it.
A Kafka cluster can spend bandwidth describing quiet partitions even when it has little new record data to replicate. Before incremental fetches, a follower's requests enumerated all the partitions it wanted, and responses included metadata for those partitions. Kafka 1.1 introduces a protocol that lets the two sides remember that information between requests. The March 28 release summary identifies KIP-227 as a partition-scalability improvement.
The problem is particularly visible when a cluster has many partitions with infrequent writes. Reducing the time between fetches increases the frequency of metadata exchange even if most partitions remain unchanged. Faster polling can therefore increase the overhead of describing the same state without increasing the amount of useful record data returned.
I regard Kafka's partition key as a central application design decision because ordering exists within the partition. Supporting many such partitions has a separate operational cost. This release addresses one component of that cost without pretending that a more efficient fetch protocol removes the consequences of the application's partitioning choices.
A session makes omitted information meaningful
KIP-227 introduces fetch sessions. A full request can establish the partition set. Later incremental requests identify the session and supply changes to that set or its fetch parameters. An unchanged partition can be omitted because the leader already knows its previous state.
The response follows the same principle. The broker includes partitions with new data or changed metadata, such as a high watermark or log start offset. A response with no partition entries can therefore mean that the session has no new information to report. It does not mean that the client has abandoned all its partitions.
This changes how request contents must be interpreted. A full request describes the collection explicitly. An incremental request describes changes relative to a remembered collection. Someone inspecting the latter without its session history cannot reconstruct the entire requested partition set from that one message alone.
The 1.1.0 FetchSessionHandler implementation makes the distinction visible in code. It retains the current session partitions and builds separate collections for entries to send and entries to forget. Its incremental builder compares the next requested partition data with the retained values and omits entries that have not changed.
That implementation detail explains why omission needs an established session. Without retained state, leaving a partition out would discard its meaning. With retained state, the same omission reduces the message size while preserving the existing request. The optimization depends on both sides agreeing which collection the session represents.
Epochs detect disagreement about the shared state
The proposal pairs a session identifier with a fetch epoch. An identifier distinguishes the session on the receiving broker, while the epoch advances with requests. The broker expects the next epoch after processing the preceding request. An unexpected epoch signals that the request sequence and retained state may no longer agree.
The protocol also defines an error for a session identifier the broker does not know. A client cannot assume the remote cache preserves its entry indefinitely. The client implementation recognizes the missing-session condition and returns to initial fetch metadata so it can establish the relationship again.
These are correctness requirements introduced by retaining state, rather than incidental error messages. A smaller request depends on what an earlier request established. Detecting a lost or invalid session prevents the broker from interpreting a delta against the wrong baseline. The full-request path provides the way to describe the collection again when incremental state is unavailable.
There is also a fairness concern when a response size limit prevents returning every available partition's data. KIP-227 specifies rotating the server's partition order after returning data for an entry. Since the client need not list every partition in an incremental request, responsibility for that rotation cannot depend entirely on reordering the next request.
Network savings require bounded memory
Remembering each fetcher's partition data consumes broker memory. The proposal therefore limits the number of incremental sessions using max.incremental.fetch.session.cache.slots, with a default of 1,000. It describes eviction rules that favor follower sessions over ordinary consumer sessions and prioritize useful retained state while limiting cache churn.
This is the tradeoff behind the reduced metadata exchange. The old messages carried a more complete description on each interaction. The new protocol can send less by keeping a description on the broker. A session-cache limit makes that retained state finite, so eviction and reestablishment are normal parts of the design.
The proposal adds metrics for session count and cached partition count, plus an eviction rate. Those measurements describe whether the cache is holding the state the optimization needs. They provide more direct evidence about this mechanism than aggregate record throughput alone, which can change for many unrelated reasons.
I find that accounting useful because the claimed improvement is specific. Quiet partitions need not repeatedly occupy the wire with unchanged metadata. The implementation still pays for retaining their session state, and a lost entry requires the full description again.
Streams recovery has separate maintenance work
The 1.1.0 release notes also list corrections around state restoration and standby replicas. KAFKA-6214 addresses a Streams crash when standby replicas accompany an in-memory state store. KAFKA-6269 addresses KTable restoration after a rebalance, while KAFKA-6205 concerns restoring stores before initializing the topology.
Those entries concern application state management, distinct from broker fetch-session caching. A session remembers protocol information used to retrieve records. A Streams state store represents processing state derived from records. Both involve retained information, but their lifetimes and recovery responsibilities differ.
The release consequently improves two different parts of operating a retained log and its consumers. Incremental fetches reduce repeated partition descriptions during ongoing retrieval. Streams fixes address particular failures while reconstructing processing state. For the fetch change, the key practical difference is that a quiet partition remains part of a valid session without needing an unchanged entry in every request and response.