Etcd 3.6 makes storage compatibility an explicit operation

Replacing a database executable with an older version is not a complete rollback procedure. The newer executable may already have changed the format of persistent state. Etcd 3.6 makes that distinction explicit by supporting a downgrade operation that migrates the schema before operators replace cluster members with older binaries.

The May 15 release announcement describes the first minor release since 3.5.0 on June 15, 2021. Its changes include a supported downgrade path and a transition in where cluster membership information is authoritative. Memory improvements are also substantial, but the storage transition is the part that most directly changes the meaning of an operational rollback.

I value Kubernetes for continuing to run workloads when control plane components are unavailable. That property does not make the control plane datastore incidental. Etcd is where the persistent state supporting the API needs its own carefully defined upgrade and recovery behavior.

An older process needs an older schema

The announcement states that old etcd binaries cannot open newer data schema versions. A 3.5 binary cannot start with data created by 3.6. This is a compatibility rule for persisted state, so returning to an earlier container image does not by itself return the database to a state that image understands.

Etcd 3.6 introduces full downgrade support through work spanning the 3.5 and 3.6 branches. At a high level, the cluster first migrates its data schema to the target version. A rolling replacement of processes follows that migration. This sequence turns compatibility into an explicit state transition that the cluster performs before older code takes responsibility for the data.

The release announcement gives two distinct commands. Validation checks the requested target before downgrade mode is enabled. These examples assume an existing etcdctl connection configuration with the correct endpoints and credentials.

etcdctl downgrade validate 3.5

If validation succeeds and the documented preparation is complete, enabling downgrade starts the schema migration. It is a mutation of the cluster's operating state, not another inspection command.

etcdctl downgrade enable 3.5

The 3.6 downgrade guide explains that enabling downgrade also changes the cluster protocol and available features to the target version. Operators confirm that each member's storage version has migrated before proceeding with binary replacement. This is why a process version and a storage version are useful as separate observations.

During a rolling transition, different executable versions coexist. The cluster has to agree on behavior that those members share. The protocol transition and schema migration establish that common behavior before the last new executable disappears. A rolling restart without those steps would provide no equivalent compatibility guarantee.

Membership moves to the v3 store

The other storage transition concerns an older internal dependency. The announcement says the v2 store has been deprecated since 3.4 but remained the source of truth for membership information. In 3.6, the v3 store becomes the sole source of truth for that information.

The enable-v2 flag is removed, so users can no longer turn on the legacy v2 API through that switch. However, the announcement explicitly says the v2 store still exists in 3.6. Describing the release as complete removal of the legacy store would therefore be wrong. The authority for membership changes now, while additional removal work remains.

There is also a startup condition that makes old data relevant. Etcd 3.6 refuses to start when the v2 store contains data other than membership information. An installation can therefore encounter a compatibility problem even if its current clients have already moved to the v3 API. Historical data still present in the old store matters to the new binary.

The announcement identifies the check v2store command in etcdutl from 3.5.18 onward as the migration aid. Its purpose is to verify that only membership information remains there. That check answers a concrete storage question, independently of whether a current application still sends v2 API requests.

This is a useful example of why deprecating access and removing internal representation are separate operations. Stopping new use of an interface does not erase existing persisted state or the implementation's own historical dependency on that state. The release makes progress on both concerns without claiming they have become the same task.

Retaining less Raft history changes memory consumption

The release reports a reduction of at least half in average memory consumption in its comparisons. According to the announcement, a lower default snapshot-count and more frequent compaction of Raft history account for most of the result. The default count falls from 100,000 in 3.5 to 10,000 in 3.6.

Those figures describe a change in how much history the implementation retains between snapshots. They do not describe deleting ninety percent of application keys. The announcement's discussion concerns Raft history, which is distinct from the logical key-value data that a client expects to read.

The published memory charts compare 3.5.20 with 3.6.0-rc.2 under several read/write ratios. That provenance limits the percentage claim to the reported comparisons. It is useful evidence about the implementation change, without establishing an identical saving for every existing Kubernetes datastore or workload.

Throughput measurements in the same announcement show an average improvement of approximately ten percent. The project attributes that to multiple smaller changes, including free-page query optimization. It does not present one new algorithm as the sole cause. Keeping the memory and throughput explanations separate prevents a lower retained-history count from becoming an unsupported explanation for every performance result.

I find the memory mechanism more informative than a headline percentage. It identifies the retained state that changed and the configuration default responsible for much of the difference. That makes the result easier to relate to an actual deployment's configuration.

Operational tools now have clearer boundaries

Etcd 3.6 separates online and offline administration more firmly. Etcdctl interacts with a running service over the network, while etcdutl operates directly on data files. Several overlapping operations leave etcdctl as part of that cleanup.

Snapshot status and restore now belong to etcdutl. Offline defragmentation also uses etcdutl, while etcdctl defrag remains an online operation. A script that treats the two utilities as interchangeable will therefore need a specific change at the operation it performs. Renaming the executable everywhere would be equally incorrect because network operations still belong to etcdctl.

Health endpoints receive a similar clarification. The announcement describes livez as an indication that the instance is alive and readyz as an indication that it can serve requests. These endpoints were also backported to older supported releases. Their availability should therefore be checked against the actual patch version, without assuming that every operational improvement first becomes usable with the 3.6 minor upgrade.

Peer and client endpoints also become more distinct. Earlier versions allowed peer endpoints to handle client requests through an implementation oversight. Version 3.6 stops doing so. An application using a peer endpoint as a client endpoint can fail after upgrade even though the service is otherwise healthy.

The change removes accidental behavior, but accidental behavior can still have callers. Its operational significance is the need to identify the purpose of an endpoint in configuration. A reachable address is not sufficient evidence that it is the supported address for that kind of request.

Correctness evidence includes failures during operations

The announcement describes robustness testing that generates traffic while injecting failures. The test records requests and responses and then checks linearizability. It also checks watch API guarantees. This examines whether the history observed by clients remains consistent with the database's promised behavior under failure.

That method connects to the release's discussion of earlier consistency bugs. The announcement records failures involving a crash during data application or defragmentation, and a durability issue in a single-member cluster. Their fixes were backported before this minor release. They should not be read as newly discovered vulnerabilities that require 3.6 as the only repair.

The significance is the kind of evidence the project now collects. A process restarting successfully does not establish that acknowledged operations survived or that revisions remain consistent. Recording client operations across the failure boundary lets a checker evaluate the database contract itself.

The release also introduces Kubernetes-style feature gates in place of an approach that changed flag names when experimental features stabilized. That gives future feature transitions a more explicit lifecycle. Alongside schema-aware downgrade and the move of membership authority, it makes version transitions something the implementation represents directly. The documented sequence remains essential because neither a healthy process nor an older image can establish storage compatibility on its own.