CockroachDB incremental backups could miss data written into the past

An incremental backup assumes it can identify the data added since its preceding boundary. CockroachDB's bulk-write path complicates that assumption because it can install data with timestamps earlier than the previous backup. A later incremental scan can therefore miss data that arrived after that backup completed.

The April 30 advisory describes possible data loss after restoring affected incremental backups. It identifies index creation, RESTORE and IMPORT as operations that can overlap a backup and produce the omission. Full backups are unaffected.

The advisory follows a preliminary notice on April 6. Cockroach Labs says its engineers found the problem through internal investigation and, at publication, it knows of no affected user. That statement concerns observed incidence. The disclosure still identifies stored backup chains and restored databases that may be invalid.

I value CockroachDB's consistency guarantees, but a backup is a separate representation of database history. Replicating the live data correctly does not prove that an incremental export captured it. The patch descriptions make that distinction concrete.

Arrival order and data timestamps can differ

The 20.2 backport for index backup selection explains that AddSSTable requests can write into the past. Incremental backups perform a time-based scan from the previous backup. Data installed with an earlier timestamp can fall outside that scan even though it was not present in the earlier backup's output.

For example, one backup can finish while a table remains offline. A bulk write can subsequently install an older timestamp, and the next backup can begin its scan after that timestamp. Both backup intervals can appear to cover the elapsed time while neither contains that write. The patch description identifies precisely this mismatch.

This is a different mechanism from losing a backup file or failing to upload it. The incremental backup can complete its chosen scan while the scan boundaries omit necessary data. A successful job result therefore does not establish completeness for the affected sequence.

Index backfills expose the problem while a new index is being constructed. The fix restricts backup to PUBLIC indexes and excludes ADDING indexes. That prevents a backup from treating an incomplete index as a stable object whose future entries can be recovered through the ordinary incremental boundary.

The relevant distinction is the index's state during backup, not simply whether the final schema contains the index. A backup chain can cross the interval in which its data is still being populated. The later public state does not retroactively insert missed entries into an earlier layer of the chain.

This explains why examining only the current schema is insufficient to assess older backups. The advisory asks about operations in progress when those backups ran. The backup's history matters even after the live database has finished the operation.

Restored and imported tables need another capture

The backport for tables coming online describes the corresponding issue for RESTORE and IMPORT. These operations can write through AddSSTable while a table is OFFLINE. A following incremental backup can miss those writes because their timestamps precede its lower boundary.

The fix captures all the table's data again in the incremental backup that brings the table back online. That gives the backup chain a complete copy of the data associated with the transition. It avoids relying on the ordinary incremental time scan for those bulk writes.

The patch description states the performance cost. An incremental backup in which an import or restore completes can become much slower because it captures the entire table again. That extra work follows from the correctness repair.

This is a useful qualification when interpreting backup duration after updating. A longer incremental run can reflect the new obligation to recapture data. The release does not claim that every incremental backup remains equally small after a bulk operation.

The patch mentions planned future work to narrow that recapture. As of this advisory, the implemented behavior is the complete capture described in the backport. A possible later optimization does not change the cost operators should expect from the current fix.

Older backups require restore-time handling

Preventing new omissions does not rewrite backup layers already stored. The index revalidation backport addresses part of that retained risk. It identifies indexes that were offline or being built in any layer of the backup chain and validates those restored to a public state.

If validation finds missing entries, the index is marked offline for reconstruction after RESTORE. That supplies a way to detect and correct certain incomplete indexes. It does not establish a general repair for every possible omission of imported or restored table data.

The advisory uses qualified language for that reason. The fixes can detect invalid indexes and correct them automatically in some cases, with a possible performance impact. That is narrower than promising that all earlier invalid backup chains become safe after installing the update.

A separate backport for restored bulk jobs restores IMPORT and RESTORE jobs in a reverting state. Its description explains that their nontransactional writes cannot be trusted to appear in every backup. Reverting lets those jobs attempt to clean up the untrusted writes.

The new behavior therefore acts at several points. Backup selection avoids incomplete indexes. Capture after an offline-table transition includes the required data again. Restore-time validation and job state address consequences retained in some older backups.

I find these separate repairs more informative than a broad claim that backup is fixed. They identify which state each correction handles. They also explain why a software update and confidence in an existing backup chain are separate outcomes.

The advisory's fallback is a fresh full backup

For users unable to upgrade, Cockroach Labs recommends taking a full backup after index work, IMPORT or RESTORE completes. It warns against relying on incremental backups taken during those operations. If the relevant history is uncertain, it recommends treating only full backups as reliable.

That recommendation follows the mechanism directly. A new full backup after completion does not depend on an earlier incremental boundary covering data installed during the operation. The advisory explicitly says full backups are unaffected by this issue.

The backup guide available at the advisory date distinguishes full backups from incremental chains. Restoring an incremental chain requires its preceding full and incremental backups. A chain is therefore a sequence of required data, not a collection of independent substitutes.

The guide also explains why incremental backups exist. They reduce storage and backup work for clusters too large for frequent full copies. The advisory changes when that efficiency can be trusted on affected releases, especially around incomplete bulk operations.

For a production database already restored from a chain overlapping a secondary-index backfill, the advisory suggests recreating those indexes and dropping the old ones. That addresses index contents. It should not be expanded into a claim that rebuilding indexes reconstructs missing base-table data.

Fixed releases and the affected history

The advisory says the problem affects CockroachDB 19.1 and later. It names 20.1.15 and 20.2.8 as fixed maintenance releases. The 20.2.8 release notes, dated April 23, list the missing-backfill and offline-table fixes.

Those dates distinguish release availability from the full advisory. April 30 is the date of this disclosure, with the relevant 20.2 maintenance fix already available. The advisory also names a forthcoming major-version fix, but the released maintenance branches provide concrete upgrade targets.

Incremental backups are an enterprise capability at this point. Full backups are available more broadly, and the advisory says they are unaffected. That scope matters when identifying which deployments could have created a problematic chain.

An affected history requires more than merely having used the database version. The backup sequence must overlap the specified incomplete operations in a way that misses their data. Uncertainty about that sequence is itself part of the advisory's guidance, since operators may lack enough history to rule out the overlap.

The immediate repair for new backups is updated code. The retained-state response concerns backup chains already at rest and databases already restored from them. A fresh full backup after completed bulk work and reconstruction of suspect secondary indexes address different parts of that history, according to the published mitigation.