CockroachDB 22.1 upgrades can leave changefeeds emitting only progress

A progress message is useful only if the work it summarizes has actually happened. CockroachDB advisory A-82309 describes changefeeds continuing to emit resolved timestamps while emitting no row data after an upgrade. A consumer watching only those timestamps can therefore conclude that a stream is current when it has missed changes.

The June 3 advisory concerns changefeeds created before an upgrade from 21.2 to 22.1.0. Delivery can stop during the rolling upgrade, and the failure may produce no log errors. Feeds created on 22.1 are unaffected, subject to the advisory's warning about creating them in a mixed-version cluster. The fix is in 22.1.1.

I value CockroachDB's explicit consistency guarantees, but a downstream stream needs its own evidence of delivery. This failure is particularly consequential because its progress channel continues to speak. The advisory explains why that channel cannot establish recovery for an affected feed.

Existing job state meets a new implementation

The original issue 82309 gives a short reproduction. Create a changefeed with resolved timestamps on 21.2, pause it, upgrade to 22.1, resume it, and write to the table. The sink receives resolved timestamps but no data messages. The reproduction isolates an existing feed crossing the version boundary.

The 22.1 repair in pull request 82312, merged June 2, changes how several components obtain target specifications. Instead of reading one target-specification field directly, they call AllTargets with the feed details. The affected uses include topic naming, row topic selection, and the row-fetcher cache.

That patch identifies compatibility with existing feed descriptions as part of the problem. A feed's persisted configuration must still identify the targets to the new runtime. Reading only the new representation can leave components without the watched targets they require, even though the job itself resumes.

The patch also makes the row-fetcher-cache constructor return an error when it cannot derive any target specifications. The caller then handles that construction error. This is a useful secondary change because an empty target set should not silently resemble a healthy data-delivery path.

A rolling upgrade produces a gradual symptom

The advisory says emitted-message and emitted-byte metrics begin falling as soon as nodes start upgrading. The decline can be gradual, depending on upgrade speed. Once all nodes run the affected new version, data emission reaches zero for affected feeds.

That timing distinguishes the problem from a single abrupt sink outage. The symptom can track the changing version mix of the cluster. A feed created before the upgrade carries configuration across that transition, while a newly created feed on the new version has a different history.

The relevant metrics are changefeed.emitted_messages and changefeed.emitted_bytes. The advisory also directs operators to observe whether the downstream sink is receiving data. These observations concern data delivery. Merely seeing the job continue or the sink accept progress messages does not answer the same question.

The advisory explicitly says resolved messages from the affected feed are incorrect and must be disregarded. That is a stronger statement than ordinary delivery lag. A lagging but truthful frontier can help locate a recovery point. This frontier cannot support that inference.

The usual restart cursor is invalid

For an already affected feed, the documented remedy is cancellation and recreation. The important choice is how the replacement catches up. A new feed normally performs an initial scan of its tables and then emits subsequent changes. That is the simplest documented way to reconstruct a current stream without choosing a historical cursor.

For large tables, the advisory offers a more targeted restart from a timestamp a few minutes before the upgrade began. The timestamp comes from the upgrade history, not the affected feed's reported high-water mark. The usual method of obtaining that high-water cursor is explicitly invalid in this case.

The advisory supplies SQL for expressing an absolute time as the cursor value. Its example uses a UTC timestamp before the upgrade boundary.

SELECT 1e9 * ('2022-06-02 13:59:00+00'::timestamptz)::int;

The literal is illustrative, not a restart point suitable for every cluster. The required value depends on when that cluster's upgrade started. Choosing it independently is necessary because the defective feed may have advanced its progress indication across changes it never emitted.

I find the independent timestamp requirement the most useful recovery detail. It identifies which evidence remains trustworthy when the feed's own progress accounting is wrong. An upgrade start time and a data-delivery frontier are ordinarily different records, and this failure makes that distinction operationally necessary.

Starting from now deliberately leaves a gap

The advisory also describes recreating a feed with no_initial_scan. That begins with new changes and skips both the initial table scan and events since the upgrade began. It is an available option with an explicit consequence. It does not recover the missed interval.

These restart choices produce different downstream contents. A full scan supplies the current table contents. A cursor requests changes from a chosen historical boundary. Starting now accepts that previous missed changes will not be supplied by the replacement feed. The correct choice depends on what the sink needs to reconstruct.

The advisory does not provide a universal command that can choose those semantics on behalf of an application. It preserves existing sink and feed options while requiring the operator to choose the catch-up behavior. That is appropriate because a state-rebuilding consumer and a history-dependent consumer can need different inputs.

Upgrade planning needs to preserve feed history

For clusters not yet upgraded, the advisory directs users to 22.1.1 or later and warns against creating feeds while nodes run mixed versions. For clusters already affected, updating alone does not replace the cancellation and recreation instructions. Prevention and recovery depend on the state the feed has already crossed.

The patch restores target interpretation and turns missing targets into an error path. Recovery reconstructs the feed with a chosen data boundary. Both steps matter because a resumed job and an advancing resolved timestamp were insufficient evidence that row changes had reached the sink.