CockroachDB restore resumption could finish with missing rows

A restore job that reports success can still have skipped part of its input. CockroachDB 22.2.6 introduced that failure when an in-progress restore resumed. Some backup data files could be omitted, leaving missing rows even though the job completed. A node restart could trigger the resume internally, so an operator did not need to pause the restore deliberately to encounter the defect.

The March 29 advisory A-98779 identifies 22.2.6 as the release that introduced the bug. CockroachDB 22.2.7 removes the incorrect optimization and restores the behavior of 22.2.5. The advisory also supplies a cluster setting that immediately mitigates the issue on 22.2.6, without waiting for replacement binaries.

I accept CockroachDB's distributed coordination costs because consistency matters to me. A restore needs an equally concrete completeness guarantee. Consensus over the restored writes cannot compensate for input files that the restore job never selected.

Resuming is part of ordinary execution

Manual pause and resume is one route into the faulty behavior. The advisory also names automatic retries following node restarts and certain internal rebalancing events. These are important exposure conditions because a restore can resume as part of its normal distributed job lifecycle. A user watching only the final job status may never see the intermediate event.

The defect concerns the work selected after that resumption. It does not require a backup file to have been lost from storage or corrupted. The stated failure is that the restore skips some files it should process. The difference matters when interpreting a successful job alongside a backup repository whose files remain present.

Completion reflects the work the implementation believes remains. If that selection is incomplete, the job can finish its selected work while missing intended data. The advisory's explicit warning about missing rows after success is therefore the central consequence. It prevents a successful terminal status from being treated as independent evidence that all relevant input was consumed.

The source does not claim that every restore on 22.2.6 loses rows. It identifies a resumed restore as the relevant condition and says files may be skipped. That scope supports immediate mitigation without inventing an incidence rate or a deterministic failure for every backup chain.

Restore planning connects spans to files

The restore-span implementation at 22.2.7 explains the corrected path's job. It partitions required key spans into restore entries and assigns overlapping files from the supplied backups. Before generating that coverage, it trims or removes required spans according to a low-water mark representing progress.

That makes resumption a planning operation as well as a scheduling operation. The job must identify the remaining key ranges and include the backup files overlapping them. A saved progress position has meaning only when the reconstructed work still covers the data after that position. The advisory does not publish every detail of the faulty optimization, but the corrected implementation shows why span coverage is relevant to file selection.

Files can overlap a required span without matching its exact boundaries. The implementation intersects spans with file coverage and attaches a file to each restore partition that needs it. This is more involved than advancing an index in a flat list of filenames. Backup layers and key ranges must combine into the remaining restore work.

The code also explains the cost that makes such planning a performance concern. Each restore span incurs splitting, scattering, processing and flushing work. Large numbers of tiny spans increase that overhead and leave small ranges to merge later. The implementation groups work toward a target size while retaining all files that overlap the grouped ranges.

This provides context for the optimization tradeoff without assigning an undocumented motivation to the faulty change. Reducing planning or per-span overhead is useful only while preserving complete coverage. The released repair chooses the earlier behavior by removing the optimization identified as incorrect.

The immediate mitigation selects the simpler path

For 22.2.6, the advisory gives this exact command.

SET CLUSTER SETTING bulkio.restore.use_simple_import_spans = true;

Cockroach Labs says it has already applied that setting automatically to affected Cloud clusters. Self-managed operators running the affected version have the same documented mitigation available through the cluster setting. The binary correction is 22.2.7, which the advisory recommends as the upgrade target available at publication.

This setting changes restore behavior going forward. The advisory does not claim that setting it reconstructs rows omitted by an already completed restore. Prevention of a new omission and repair of an earlier incomplete result are separate tasks. A statement that the vulnerability is mitigated cannot establish that previously restored contents are complete.

My preference is to retain that distinction in recovery procedures. A completed job needs to represent the intended backup contents, and changing the algorithm for a later job does not alter the evidence for an earlier one. The advisory establishes a way to prevent the known resume defect, rather than a retrospective certificate for every restore performed on 22.2.6.

The affected unit is the restore operation

The version range is unusually narrow. The advisory identifies introduction in 22.2.6 and reversal to 22.2.5 behavior in 22.2.7. That supports a specific operational response for clusters on the affected maintenance release. It does not imply that every backup created by the branch is defective or that the original database lost the same rows.

The failure occurs while reconstructing data from backups into a restored result. Backup creation, file durability and restore selection are distinct stages, and this report concerns the last of them. Available files can remain sufficient for a correct restore even when one implementation fails to select them all.

The practical correction is therefore to use the fixed release or the documented simpler-span setting before relying on a resumed restore. Internal retries remain possible, so avoiding the manual pause command is not an equivalent workaround. The released mitigation addresses the selection algorithm that those retries reenter, including resumes an operator never explicitly requested.