Kafka 2.1 fences stale replication and adds Zstandard

A broker can lose contact with ZooKeeper while still communicating with other brokers. That partial failure is more dangerous than a clean process exit because different parts of the system can disagree about the broker's role. Kafka 2.1 strengthens replication fencing to address that condition. The November 20 release summary describes the risk as divergent replicated logs and possible data loss under rare circumstances.

The same release adds Java 11 support and Zstandard compression. Those changes affect runtime and record compatibility, while replication fencing concerns whether an operation belongs to the current leadership state. Understanding the release requires keeping those different boundaries visible. A successful binary upgrade does not mean every new behavior can immediately be enabled with every existing client.

I value Kafka as retained history that several consumers can interpret independently. The consistency of that history depends on replicas agreeing which leader's log they follow. A protocol that detects stale leadership is directly relevant to the reason to keep that history in the first place.

A leader epoch gives a request a leadership context

Jason Gustafson's replication fencing patch, pull request 5661, introduces checks for a request's current leader epoch. The receiving partition compares the supplied epoch with its own. If its local epoch is newer, it returns FENCED_LEADER_EPOCH. If its local epoch is older, it returns UNKNOWN_LEADER_EPOCH.

These errors distinguish two disagreements that a generic network failure cannot express. A requester can have stale leadership information, or it can have information the receiving broker has not yet applied. The epoch lets the participants detect that they are discussing different leadership states instead of continuing as if the broker address alone established a valid relationship.

The patch applies this context to replication work and leader-epoch offset handling. Its code separates checks of the epoch from checks that a replica exists or that the local broker is the leader. Those conditions are related but distinct. A process can be reachable and possess a replica without having the leadership state required by the request.

That is the relevance to a broker isolated from ZooKeeper. Continued network communication does not establish continued authority to participate under old state. Fencing supplies an explicit rejection when the leadership context is wrong. This changes the protocol's ability to detect a partial failure, without claiming that network partitions cease to occur.

The patch also changes client error handling to refresh metadata for the new epoch errors in applicable fetch paths. That is an example of why adding a server-side error requires a corresponding interpretation in the surrounding machinery. A rejected operation needs a recovery action that addresses stale state, rather than an assumption that the socket itself has failed.

Zstandard changes what downstream readers must understand

The 2.1 upgrade guide states that both brokers and clients must be upgraded to use Zstandard. Consumers older than 2.1 cannot read topics using that compression. The capability therefore has a downstream compatibility boundary beyond the producer choosing the codec.

For a Kafka 2.1 producer, the codec selection is a short configuration entry. It is suitable only when the relevant broker and consuming clients support the compressed records.

compression.type=zstd

The release summary describes Zstandard as offering compression comparable to gzip with higher compression and especially decompression speeds. That is the project's description of the codec, not a measured prediction for every workload. The practical change in Kafka is that the codec becomes an available representation that participating readers must understand.

A shared topic can have more consumers than the producer's immediate application. Enabling a representation they cannot decode changes the usefulness of the retained records to those consumers. The topic's compatibility inventory therefore needs to include its downstream readers, not merely the client that first writes the data.

I find this a useful consequence of Kafka's independent-consumer model. Independence lets several systems consume the same log, but it does not make their supported record encodings irrelevant. A compression decision belongs to the contract shared by everyone reading that topic.

The upgrade has a point beyond binary replacement

The upgrade guide describes keeping the previous inter-broker protocol while replacing broker binaries one at a time. At that stage, it says downgrading remains possible. Enabling the 2.1 protocol is a separate step after the binaries have been upgraded.

The guide also warns that 2.1 changes the internal schema used for consumer offsets. Once brokers begin using the latest protocol, it says the cluster cannot be downgraded to an older version. This is a concrete persistence compatibility boundary. Retaining an old executable does not mean that executable can interpret state subsequently written in a newer schema.

Consumer offset retention semantics also change. The guide says an active group's partition offsets do not expire while it remains subscribed to the corresponding topic. When the group becomes empty, the configured retention period applies. Standalone consumers outside group management retain the separate rule based on time since the last commit.

These details concern the position from which consumption can resume. They are separate from how long the topic retains its records. Preserving the records and preserving a consumer group's committed position are related operational goals with distinct stored information and retention behavior.

The release also introduces a producer delivery timeout with a default of two minutes and raises the default retry count to Integer.MAX_VALUE. Its upgrade guide directs users who previously reduced retries to fail faster toward the new delivery-timeout control. The intended bound becomes total delivery time rather than an arbitrary count of retry attempts.

For a deployment, these changes create several identifiable decisions. Broker leadership checks prevent replication requests from proceeding under the wrong epoch. A codec choice determines which readers can decode the resulting topic data. The protocol transition affects persisted offset compatibility, while the producer timeout bounds how long delivery may remain pending. Each can be described from the actual request or stored state it governs.