CockroachDB changefeed checkpoints could skip unscanned data
Advisory A-123371 explains how restart checkpoint restoration could advance progress for spans that had not been scanned. A completed initial scan could therefore leave consumers with missing rows.
Reducing duplicate events during restart must not turn unfinished work into completed progress. CockroachDB advisory A-123371, published June 17, describes an initial-scan bug where that distinction failed. A restarted changefeed could advance progress for spans it had not scanned, leaving consumers with incomplete data even when the initial scan subsequently reported completion.
The defect originated in code intended to reduce duplicate messages after restart. Persisted checkpoints let a resumed job skip portions already scanned. Another part of progress initialization could interpret that checkpoint too broadly, forwarding every span tracked by a node to the lowest checkpoint timestamp. Some of those spans still represented unfinished initial-scan work.
I value CockroachDB's consistency guarantees, but a consistent source database does not itself prove that a downstream copy is complete. A changefeed has its own progress and restart machinery. This advisory identifies how that machinery could discard work without a corresponding failure of the database's committed rows.
The initial scan has unfinished spans and completed spans
The fix in pull request 123625 explains the relevant state. Initial resolved timestamps had previously remained zero until the initial scan completed. Loading checkpoint timestamps on restart introduced a mixture of completed progress and spans still carrying zero initial progress. Initialization needed to preserve that distinction.
The fix initializes the initial high-water mark and frontier as zero if any initial resolved timestamp remains zero. That prevents a nonzero checkpoint for some work from becoming a claim about all work. A progress frontier must remain constrained by unfinished spans, even if another span has reached a later point.
This is the central correctness issue. A checkpoint describes progress over a particular subset of the job. It cannot safely be treated as a global lower bound unless every relevant span has reached it. The advisory's failure occurs when restoration effectively makes that broader claim and the restarted scan no longer emits some required data.
Restoring progress therefore requires retaining both the per-span checkpoint and the fact that some spans have no completed initial scan. Collapsing those states into one nonzero timestamp discards information needed to decide what may be skipped.
The order of span processing contributed to the bug. The advisory requires the final span in the aggregator's nondeterministic order to be fully enclosed by the latest checkpoint. Different processing orders could therefore produce different outcomes from the same broad restart situation. A restart alone was not a sufficient condition for every affected feed to lose events.
The trigger requires a checkpoint and an interrupted scan
The affected changefeed must use initial_scan = 'only' or initial_scan = 'yes'. It must restart after writing a checkpoint but before finishing the initial scan. A transient error or a participating node restart can cause that interruption. The processing-order condition then determines whether the faulty initialization path advances unfinished work incorrectly.
That scope distinguishes the advisory from a claim that all ongoing changefeeds lose changes. The initial scan and its resumed checkpoint are essential parts of the mechanism. A report about steady-state streaming without an initial-scan restart would need separate evidence before attributing its symptoms to this bug.
Several factors increase the likelihood of encountering the required state. They include long initial scans, large tables with many ranges and feeds targeting tables with substantially different row counts. More opportunities for checkpointing during an unfinished scan create more opportunities for a restart to restore mixed progress.
The advisory also identifies low values of changefeed.frontier_checkpoint_frequency or changefeed.frontier_highwater_lag_checkpoint_threshold as contributing factors. On 23.2 or later, enabling changefeed.shutdown_checkpoint.enabled increases likelihood too. These are contributing conditions, not independent proof that a particular consumer missed data.
I find that distinction important for interpreting operational evidence. A configuration can identify exposure, while completion counts and restored progress help identify what happened to a specific feed. Conflating those questions would either overstate loss in every affected deployment or overlook a feed that continued running after the omission.
Completion can be consistent with missing rows
One documented symptom is an initial scan that completes after emitting fewer rows than the targeted tables contain. Another is cr.node.changefeed.backfill_pending_ranges dropping abruptly to zero on some nodes without a corresponding increase elsewhere. Both symptoms follow from progress being advanced without all corresponding work occurring.
The metric alone does not establish a complete downstream dataset. In this defect, the state used to describe remaining work is part of what becomes incorrect. A reassuring progress display can therefore agree with the broken bookkeeping while disagreeing with the actual emitted data. The advisory supplies the row-count symptom as separate evidence of incompleteness.
This also explains why duplicate reduction deserves careful treatment. Re-emitting already delivered initial-scan data is a different failure from omitting data never delivered. The optimization attempted to avoid repeated work after restart, but the checkpoint interpretation needed to preserve which work had actually completed. Skipping a span changes the consumer's available information permanently unless some later repair supplies it.
The published fix spans several release branches
The advisory affects all 22.2 versions, 23.1.0 through 23.1.21 and 23.2.0 through 23.2.5. It also includes 24.1 testing versions through 24.1.0-rc.1. The fix is listed in 23.1.22, 23.2.6 and 24.1.0-rc.2, and the advisory recommends upgrading to repaired releases, including 24.1.0 where appropriate.
The 23.2.6 release notes, dated June 11, describe the same initial-scan checkpoint restoration repair. The June 17 advisory provides the fuller trigger conditions and symptoms. This chronology separates availability of the maintenance fix from publication of the detailed advisory.
Installing repaired code prevents the incorrect progress forwarding described here. It does not, by itself, demonstrate that an earlier consumer already received every row. That follows from the failure mechanism, since the omitted events are downstream data, while the patch changes future progress restoration. A downstream dataset's completeness remains a question about the actual data emitted by the affected initial scan.