A CockroachDB upgrade could remove metadata needed by TTL jobs

A background deletion job can remain scheduled after the metadata it needs has disappeared. CockroachDB 23.1.9 introduced that mismatch for clusters upgrading from 22.2.x with row-level TTL enabled. The upgrade erroneously removes a table-metadata field associated with the feature. Scheduled TTL jobs then encounter the incomplete state, and their failures can crash nodes.

The September 12 advisory A-110363 defines the affected path using the starting version, destination version and use of TTL. All conditions matter. The recommendation does not apply to a cluster without row-level TTL, or to one already on the 23.1 branch using TTL. The problem belongs to this upgrade transition rather than to every execution of a TTL job on every version.

I value CockroachDB's explicit state model because it makes distributed behavior easier to reason about. An upgrade needs to preserve the background work attached to that state as well as the rows visible to application queries. Here the job survives but loses a piece of the table description it expects.

TTL expiration and deletion are different stages

The 23.1 row-level TTL documentation describes expiration as making rows eligible for deletion. A scheduled background job later finds those rows and deletes them in batches. Passing the expiration time does not mean a row immediately disappears at that instant.

This separation explains how a metadata defect can surface after an upgrade appears to have completed. The job has its own schedule, and the table's TTL configuration must remain usable when that schedule next causes work to run. A successful foreground query immediately after the upgrade does not exercise every future background execution path.

The TTL mechanism issues selection queries at a historical timestamp, obtains rows eligible for deletion and performs batched DELETE operations. The documentation describes rate limiting and lower admission priority for the deletion work. Those controls balance cleanup with foreground application traffic. They depend on a working job, rather than replacing its need for valid metadata.

TTLs can be defined through an expiration expression or a fixed interval. The fixed-interval form uses a hidden expiration column, while the expression form can refer to a suitable existing timestamp column. The advisory itself does not limit the affected upgrade to one of those forms. Narrowing exposure to only a particular table-definition style would require evidence the source does not provide.

Metadata links a table to its background behavior

The advisory says enabling row-level TTL adds a field to table metadata and schedules periodic deletion jobs. During the affected upgrade, that field is removed erroneously. The fix prevents the deletion of the field. This is a schema-metadata preservation issue in the upgrade path, not an announcement that the scheduler should stop deleting expired data.

A scheduled job and a table descriptor are separate pieces of state with a relationship. The job needs enough information to interpret the target table's TTL behavior. Preserving only the schedule is insufficient if the descriptor no longer indicates the feature state required by execution. The advisory identifies the resulting crash after that relationship breaks.

This is also why retrying an ordinary application query does not describe a remedy. The damaged state belongs to the table metadata used by the background feature. The advisory does not supply a manual catalog-editing repair. It directs users who may already have encountered the problem to Cockroach Labs support.

The consequence is broader than delayed cleanup alone. A job failing to delete expired rows might ordinarily suggest a retention backlog. The advisory explicitly says this failure can crash nodes. That moves the issue into availability planning for the upgrade, while preserving the distinction between expired-row accumulation and the reported process failure.

The recommended destination is not the newest listed patch

For TTL-enabled clusters still on 22.2.x, Cockroach Labs recommends avoiding 23.1.9. It offers 23.1.8 as the available destination or waiting for 23.1.10. At the advisory date, the fix is described as forthcoming in 23.1.10. It would be inaccurate to present that release as already available based only on this source.

The recommendation is specifically about crossing the major-version boundary. For a TTL-enabled cluster already on 23.1.x, the advisory says the recommendation does not apply. This illustrates why a destination version alone is sometimes insufficient to determine upgrade risk. The state transformation performed from a particular starting branch can be the faulty operation.

My preference is to retain the source and destination together in an upgrade record. A current version says which code is running, while the path into that version explains which metadata transformations ran. That distinction is relevant here because the advisory conditions exposure on coming from 22.2.x.

Inspection has separate levels

The TTL documentation offers SHOW SCHEDULES for viewing scheduled work and SHOW JOBS for examining job execution. These answer different questions. A schedule can exist while a particular execution fails, and a table can still accept application queries while its scheduled deletion path has invalid configuration.

SHOW SCHEDULES;
SHOW JOBS;

These commands are observation tools, not the published repair for A-110363. The advisory's preventive action is selecting the recommended upgrade path. Treating a visible schedule as proof that the table's required TTL field survived would repeat the same mistake of inferring one state from another.

The documentation also makes the deletion interval independent of the expiration definition. A TTL table carries both the rule that determines eligibility and scheduling information for cleanup. The metadata must preserve a coherent configuration across upgrade so later jobs can continue applying that rule. Changing when a job runs does not supply the missing table field identified by this advisory.

For an affected cluster that has already crossed to 23.1.9, the advisory gives a support path. For a cluster that has not yet crossed, it gives a version choice that avoids the defective transformation. Those are different operational positions, and the publication keeps them distinct while the next maintenance fix is still pending.