CockroachDB disables an asynchronous bulk flush path that lost errors
Advisory A-144650 describes bulk operations that could report success with missing data. The fix disables a path whose errors could be consumed during reset before the caller checked them.
An asynchronous task can finish with an error while the operation waiting for it still reports success. CockroachDB's April bulk-ingestion advisory describes such a failure, caused by an error being consumed during an internal reset before the top-level flush could observe it. The result could be a successful bulk operation with missing data.
Advisory A-144650, published April 28, identifies IMPORT, MOLT Fetch without verification, physical cluster replication, CREATE TABLE ... AS, and materialized-view refreshes as affected paths. It explicitly excludes restore. Index backfills also avoid the silent-success outcome because post-ingestion validation detects missing data.
I value CockroachDB's consistency guarantees and accept their coordination cost. This finding concerns the completion reporting around ingestion, where the system has to preserve errors across internal stages. A successful job status cannot compensate for a failure that never reaches the code deciding that status.
Reset consumed the result before Flush checked it
The fix's patch and explanation describe a flush crossing a range boundary and starting a goroutine for the RPC. Earlier behavior propagated the goroutine's error through the top-level Flush call. Another change made Reset wait for asynchronous tasks, assuming reset occurred only during cleanup.
That assumption was incomplete. flushIfNeeded also called Reset after doFlush, so reset could consume the error before the later flush call. The patch disables asynchronous flushing in that path. Its explanation says this does not remove an existing performance benefit, because reset already waits and makes the operation effectively synchronous.
The important defect is the error's ownership across lifecycle stages. Waiting for a task is not sufficient if the wait consumes a result that a later caller needs. A helper that logs an error during reset can leave the enclosing operation without the evidence required to reject completion.
Failure needed a particular combination of paths
The advisory describes a rare flush path encountering errors such as insufficient disk space or node unavailability. The silent-success outcome requires no separate error outside that path to force a retry or rollback. It affects 23.2.0 through 23.2.23, 24.1.0 through 24.1.16, 24.3.0 through 24.3.10 and 25.1.0 through 25.1.4, all 24.2 versions, and 25.2 testing versions through beta 1.
The 25.1.5 release note dates the corrected 25.1 maintenance release to April 28. An update prevents subsequent use of the defective path. It does not reconstruct rows or index entries omitted by an operation that already completed.
That distinction separates prevention from repair. Replacing code changes how a future ingestion reports failure. Establishing whether a prior operation ingested its intended input requires evidence about that operation's data, not merely evidence that the replacement binary is running.
The detector correlates errors with subsequent job outcomes
The project's detection script searches logs for asynchronous flush errors and excludes index-backfiller entries. It extracts job identifiers and timestamps, then correlates them with retry records and successful job completion.
The ordering matters. A later retry can change whether an earlier error indicates an incomplete final result. The script therefore compares the latest relevant flush error with retry times before joining candidate jobs to successful completions. It also reports errors without a usable job identifier, where that association cannot be established.
This is evidence-based triage, with a scope limited by the available logs. An analysis cannot reconstruct a job's history when those records no longer exist. The advisory directs users to analyze a debug archive on a separate machine and to involve support for suspected affected jobs.
Import verification needs to identify what was counted
The IMPORT INTO reference describes appending rows from external data to an existing table. It takes the table offline during import and returns it online when the job succeeds. It is an insert-only operation, so its intended result is tied to the source input and the table's prior contents.
A row-count comparison has to identify that starting state. Counting the table after an import into a nonempty table measures both previous and imported rows. Subsequent application writes can change the count again. The relevant evidence must describe the data at the operation's boundary if it is meant to establish what that import added.
The distinction between rows and index entries is equally important. A table can contain the expected primary records while a secondary index is incomplete. The advisory's separate treatment of backfill validation and bulk ingestion is a reason to avoid equating a successful count with validation of every access structure.
A persisted query result needs the corresponding source state
The CREATE TABLE AS reference describes storing a query result for later reuse. The new table does not continually update to match later source changes. Comparing it with today's source query can therefore reveal legitimate divergence as well as a possible ingestion defect.
The REFRESH reference likewise explains that materialized views keep stored results until refreshed. A refresh computes new stored results from the source. Verifying its completeness needs a comparison appropriate to that refresh's source state.
These operations make the time of comparison material. An apparently different result is not self-explanatory if the source changed between ingestion and verification. Conversely, a successful job indicator cannot establish equality when the defect itself bypassed failure reporting.
Physical replication can compare a common point in time
The physical-replication monitoring guide documents point-in-time fingerprints. It obtains the replicated time, then fingerprints source and standby at the same timestamp. This compares corresponding states instead of two independently advancing present-time views.
I find that a more useful completeness signal than replication activity alone. Progress timestamps describe how far a job says it has advanced, while fingerprints compare data at a defined common point. April's correction restores error propagation for future ingestion. Investigation of earlier work still needs evidence that the intended data reached the destination.