CockroachDB encryption registry rotation could make a store unreadable

A running database process can continue using an encrypted file whose decryption metadata is no longer durably recorded. The failure becomes visible only when the process restarts and reconstructs its state from disk. CockroachDB's encryption-at-rest registry had that failure during log rotation. An omitted metadata entry could leave one store unable to read an existing file after restart, even though the process had appeared healthy before it stopped.

The August 15 advisory A-106617 traces the bug to introduction of the append-only registry log in 21.2.0. The fix changes the ordering of rotation steps. It appears in 22.1.22, 22.2.13 and 23.1.8, with the 23.1.8 notes dating that maintenance release August 7. The advisory adds a specific rolling-upgrade procedure because a restart can expose already latent damage.

I accept CockroachDB's consensus overhead because it provides an explicit model for keeping consistent replicas. This incident makes the operational condition behind that model concrete. A healthy replica must still be able to decode its local files, and replacing several processes at once can remove more usable copies than a range can tolerate.

The registry is required to interpret encrypted files

The advisory describes per-file metadata needed for decryption. CockroachDB stores that information in an append-only log. As the log grows, rotation writes a snapshot of the metadata into a new replacement log file. The defect can omit the metadata for one file from that persisted replacement.

The omitted file's contents need not have disappeared. The problem is that after restart the process no longer has the information required to decrypt and read them. That distinction explains why a store can appear corrupt while the failure began in metadata persistence rather than a direct overwrite of every affected data file.

A registry snapshot must preserve the complete live set of metadata needed by surviving files. Rotation is useful because an indefinitely growing append log would retain an increasingly long update history. The replacement snapshot condenses that history into current state. Its correctness depends on capturing every entry that remains necessary when the process next opens the store.

The fix changes the order of rotation steps, according to the advisory. The public description does not identify a need for different encryption algorithms or a new key-management configuration. Replacing the key alone would not address the stated omission of per-file metadata from the registry. The corrected operation is the one that makes that metadata durable across replacement of the log.

Why the problem can remain hidden

A node encountering the bug continues operating normally until its process restarts. If the omitted file has already been removed before that restart, the missing metadata may never be needed again. The advisory says CockroachDB storage-engine files generally have relatively short lifetimes, which is why the issue is rare.

That lifetime dependency makes current health an incomplete indicator of durable recoverability. A process can retain the state it needs while its next startup would not be able to reconstruct it. A restart is therefore an exposure point for existing latent state, rather than necessarily the operation that first creates the omission.

The advisory says the most frequent symptom is a process exiting shortly after startup. It gives a Pebble invalid-table error mentioning a bad magic number as an example. That message is a symptom of failing to interpret the file. It does not establish from its wording alone that the disk physically changed the file's header.

Repeated startup attempts may encounter the same unreadable file because restarting does not restore the missing registry entry. The advisory directs operators toward decommissioning and replication when corruption manifests, rather than treating repeated restarts as the recovery procedure. The remaining healthy replicas provide a different source from which the cluster can rebuild the lost local copy.

A local store failure can become a quorum failure

The 23.1 replication architecture documentation describes a Raft group for each range. A quorum of voting replicas must agree on changes before commitment. With three voting copies, one unavailable replica can be tolerated, while losing two removes the majority needed for the range's progress.

The advisory limits this corruption to one store when it occurs. That locality makes recovery possible through healthy replicas, but it does not guarantee that several stores can fail simultaneously without affecting availability. A rollout restarting multiple affected processes close together can reveal latent failures on more than one replica holding the same range.

The relevant count is the usable voting membership for each range. A cluster-wide node count does not by itself establish which copies of a particular range remain available. This follows from the architecture's per-range Raft groups. The advisory's warning about quorum loss concerns that overlap between store failures and replicated data placement.

Non-voting replicas provide another useful distinction in the architecture documentation. They follow replicated data but do not participate in quorum. Counting all stored copies as if each were a vote would overstate the immediate consensus capacity. The recovery procedure needs the configured replica roles and healthy range state to remain authoritative.

My preference is to keep those guarantees separate when explaining the incident. Encryption protects stored data under the configured scheme, replication provides other copies, and quorum determines whether a range can continue committing changes. A metadata failure in one layer changes which replicas are usable by another layer.

Upgrade one node at a time

Cockroach Labs recommends a rolling upgrade to the fixed maintenance release. The advisory specifies upgrading nodes individually and pausing 15 minutes between nodes. That interval is a concrete instruction in the source, rather than an invented universal recovery time for every cluster.

If a node begins failing repeatedly at startup, the procedure changes. Operators should pause the rollout, decommission the corrupted node, and wait for all ranges to become fully replicated before continuing. Waiting for replication restores the available copies before another planned restart can expose another latent store failure.

Decommissioning is therefore part of restoring the cluster's fault tolerance, not merely removing an inconvenient failed process from a node list. The architecture documentation describes redistributing replicas as membership changes and using snapshots to transfer replica state. The surviving replicas are the basis for replacing data that the failed store cannot read.

Snapshots also have an ordering role in recovery. The documented replication process transfers a copy at a point in time and then replays later Raft-log entries to bring the recipient up to date. Beginning that transfer is insufficient to establish a usable replacement copy. The advisory's condition is full replication before the rollout resumes.

Patching and recovering answer different questions

The fixed code prevents the rotation-ordering defect in later operation. It does not establish that every file on an already affected store can become readable without rebuilding that store. The advisory explicitly warns that restarting an encrypted store may reveal latent corruption, which is why deployment of the repair includes a recovery-aware sequence.

The affected range starts with 21.2 and includes 22.1 releases through 22.1.21, 22.2 releases through 22.2.12, and 23.1 releases through 23.1.7 when encryption at rest is enabled. The advisory recommends the respective fixed maintenance branches. Deployments without encryption at rest are outside the described issue.

Those conditions are necessary for an accurate explanation. A general statement that every CockroachDB restart can corrupt storage would erase both the configuration scope and the distinction between creating and exposing latent damage. The documented problem belongs to the encrypted-file registry and the listed releases.

The maintenance release describes the same rollout risk

The 23.1.8 notes also say a restarted node with this corrupted state may fail to rejoin and that multiple affected nodes during a rollout can cause quorum loss. That independent release entry connects the advisory's mechanism to the shipped correction. The later advisory provides the more detailed registry explanation and the recovery steps.

Together the sources establish a specific operational requirement. Use the fixed implementation, restart nodes individually, and interrupt the rollout if an unreadable store appears. Before another node is restarted, the cluster must recover its replicated state through the process named in the advisory. The next restart then occurs with the restored set of healthy copies, rather than accumulating unavailable replicas across the rollout.