June 18, 2018·4 min read·engineeringcockroachdbupgradesoperations

CockroachDB 2.0.3 can hold the cluster version during upgrades

A new setting separates replacing node binaries from advancing the cluster version, preserving a deliberate point at which an operator can retain downgrade compatibility.

Replacing the last executable in a rolling upgrade and accepting the new cluster version are different decisions. The first determines which software runs on each node. The second determines which version the cluster commits to using. CockroachDB 2.0.3 adds a setting that lets an operator keep that second decision pending.

The June 18 release notes introduce cluster.preserve_downgrade_option. Its purpose is to preserve the downgrade option during a subsequent rolling upgrade. This is a control introduced in the current maintenance release, not evidence that the next major release is already available. It gives existing installations a way to prepare for a more deliberate upgrade sequence.

I accept coordination costs in CockroachDB because its transaction model makes those costs part of an explicit design. Upgrade coordination deserves similar clarity. A completed binary rollout should not be mistaken for a reversible cluster-state change unless the implementation actually preserves that option. This setting identifies a specific version transition that can be held.

The setting holds a particular version

Victor Chen's backport in pull request 25811 defines the setting as a string with an empty default. Its description says it disables automatic or manual cluster-version upgrades from the specified version until reset. The validator accepts a nonempty value only when it matches the current cluster version.

That validation makes the value meaningful. It is not a general instruction to return to any version the operator types. The supplied version identifies the current point being preserved. A mismatch produces an error that reports the cluster version alongside the requested value. This prevents a configuration from claiming to preserve a state the cluster is not actually using.

The patch also adds a check inside the version transition. While the preservation setting is nonempty, an attempted upgrade fails with an explanatory error. The check covers the transition itself, which is stronger than asking a particular upgrade script to refrain from calling it. Both automatic and manual paths meet the same stored restriction. The error names the setting that prevents advancement.

The distinction between a default and an active restriction is operationally useful. An empty value permits the normal transition behavior. Setting a version records the decision to defer that behavior. Resetting the setting removes that restriction. The implementation has a concrete state an operator can inspect instead of relying solely on a note in a maintenance plan.

Holding a version is narrower than arbitrary rollback

The name refers to preserving a downgrade option, but the mechanism works by preventing advancement. It does not implement reversal of a version change that has already occurred. That distinction follows directly from the patch's validation and transition checks. An operator must establish the restriction while the version to be preserved is still current.

A binary rollback and a data-format rollback are different kinds of operation. A cluster that has begun using newer behavior may contain state an older binary cannot interpret. Holding the cluster version is intended to retain the compatibility boundary during the upgrade. It should not be presented as permission to ignore the documented upgrade path or run any older executable against any state.

My preference is for that scope to be stated plainly. The useful capability is a pause before accepting the next cluster version. Describing it as a universal rollback switch would conceal the timing requirement that makes it work. A version-preservation setting needs to be part of the upgrade sequence before the transition it blocks.

This also separates two observations that an operator might otherwise combine. Every node may be running the replacement binary while the cluster version remains held. That is an intentional state, not necessarily a rollout failure. The remaining work is a decision about the cluster transition, which should be recorded separately from the node-by-node executable changes.

Maintenance failures need accurate outcomes

The same release fixes cockroach quit returning failure even after a node had shut down successfully. It also moves that command's warnings to standard error. These changes concern the interface through which an operator learns the outcome of a maintenance action. A successful shutdown reported as failure can send automation into an unnecessary recovery path.

This is distinct from the version-preservation mechanism, but it affects the same operational setting. A rolling procedure needs to know whether a node stopped, which binary restarted it, and whether the cluster accepted a version transition. Those facts come from different components. Accurate command outcomes help keep them separate.

Version 2.0.3 therefore gives upgrade procedures a more explicit control point. The new setting validates the version being held and blocks advancement until the restriction is reset. Alongside corrected shutdown reporting, it helps an operator distinguish a completed node operation from the separate decision to advance the cluster's compatibility state.