CockroachDB 2.0.5 fixes backup iteration after deleted rows
A double iterator advance could omit a live row after a deletion marker. The August release repairs the export loop and bounds storage batch grouping for recovery.
Skipping a deleted row during a backup is expected. Skipping the next live row is data loss in the backup. CockroachDB 2.0.5 repairs an export loop that could do both because it advanced its iterator twice. The change is one removed line, but the associated test change explains why the defect matters.
The August 13 release notes identify the backup omission alongside a RocksDB batch-group limit intended to prevent excessive memory use during startup replay. Both fixes concern operations that determine whether stored state can be recovered. They deserve more attention than their small patch sizes might suggest.
I accept the coordination cost of CockroachDB's consistency model, but that model does not independently establish backup completeness. A backup has its own code for traversing stored versions and producing an export. Its correctness needs to preserve every row the selected backup semantics require, including rows adjacent to records intentionally omitted.
Two advances omit the following key
David Taylor's patch in pull request 28196 explains that the export loop manually advanced its iterator before continue. The surrounding loop already advanced it. When skipping a tombstone, the extra advance therefore also skipped the following key.
The condition in the patch concerns tombstones during a non-incremental export when the export is not retaining all versions. That qualification matters. The fix should not be described as a failure of every backup mode under every deletion. It changes a specific skip path whose job is to omit a deletion marker while continuing with the next relevant record.
An iterator's ownership rules are easy to obscure when advancement occurs in more than one place. A branch can appear locally sensible, since it intends to move beyond the unwanted item. The surrounding loop then performs its own movement. The combined behavior differs from either piece read in isolation.
Removing the branch's explicit advance restores one advancement for that iteration. The next pass then has the opportunity to examine the following key. The fix changes traversal, not the predicate deciding which tombstones belong in the export. That distinction explains why the same selection semantics can produce a more complete result after the patch.
The test needs surviving neighbors
The regression test previously deleted all rows in its test table. Taylor changes it to delete a subset selected by a modulo condition. The expected source and restored row counts then retain three quarters of the original rows. That gives the test live records whose preservation matters after a deletion marker.
A test that expects an empty result cannot demonstrate that neighboring live rows survive the skip path. Deleting a subset creates the mixture needed to observe the defect. The expected count connects the traversal behavior to an externally visible restore result. The test therefore exercises the recovery consequence instead of merely checking that the iterator does not crash.
The broader lesson for this particular bug is about the shape of the fixture. The relevant state contains both omitted records and retained records. A fixture containing only one category leaves the transition between them untested. That transition is exactly where the extra advancement changed the result.
My judgement is that the test adjustment is as informative as the code deletion. It shows what evidence was missing from the earlier case. The corrected backup loop and a fixture with surviving rows together make the failure and its repair understandable without assuming an undocumented production incident.
Recovery cost depends on write grouping
The batch-group patch in pull request 28009 addresses another recovery problem. It limits groups of RocksDB batches to a one-mebibyte target. A larger individual batch remains allowed, but it is not grouped with other batches. The stated failure was rare multi-gigabyte groups that could exhaust memory during log replay at startup.
This is a grouping limit, not a universal limit on transaction size or server memory. Combining several writes can create a larger unit than any one caller submitted. The unit also matters later when recovery reads the log. An optimization during normal operation can therefore impose a recovery cost after a restart.
Keeping a large individual batch separate preserves that distinction. The implementation does not need to reject the batch solely because it exceeds the grouping target. It prevents additional batches from making that recovery unit still larger. The patch's size accounting gives the grouping decision a concrete bound.
For operators, the two fixes affect different recovery evidence. The backup repair prevents a particular live-row omission during export. The grouping repair reduces a particular source of memory pressure while replaying the local storage log. Neither replaces the other, since a complete external backup and a restartable local store depend on separate paths through the database implementation.