CockroachDB 26.2.3 fixes a silent tenant backup omission

CockroachDB 26.2.3 fixes a backup that could finish without capturing all the intended tenant data. The June 26 release notes say BACKUP TENANT could silently omit the last table in the tenant's keyspace when that table followed one excluded from backup. The failure involved the relationship between an intentional exclusion and data that should still have been included.

This is different from a backup job returning an error or a storage file becoming unreadable. A silent omission challenges the assumption that a completed job contains everything its scope required. The release note is brief, but it identifies a specific boundary case that can inform a meaningful regression test. The test needs an excluded table followed by a final table whose rows must remain in the backup.

I value CockroachDB's distributed consistency guarantees and accept their latency cost when the workload needs them. Recovery still needs separate evidence. The database agreeing on its live state does not prove that an export of that state captured every required table. This patch is a useful example of why backup completeness belongs in the definition of recoverability.

Excluding data has a legitimate purpose

The backup documentation describes exclude_data_from_backup as a table storage parameter. It can exclude high-churn row data whose garbage-collection needs differ from the backup schedule. The backup retains the table's schema while omitting its rows. That is the intended result for the marked table, not a sign that its backup is defective.

For an existing table whose data is deliberately disposable, the documented operation has the following form. Here transient_data is an illustrative existing table whose rows have been explicitly excluded from the recovery requirements.

ALTER TABLE transient_data
SET (exclude_data_from_backup = true);

The backup process does not contact the relevant storage ranges to retrieve that table's data. The documentation also says the setting prevents the cluster or database backup from extending the garbage-collection retention on its key span. This provides the intended benefit for data that does not need to survive recovery.

The defect is that an exclusion could affect a different table at the end of the tenant keyspace. Its data was not the intentional omission. That makes the adjacency condition important when constructing a reproduction. An example with a single ordinary table or with no exclusions does not preserve the failure described by the release notes.

A checksum cannot explain data that was never captured

CockroachDB's validation documentation distinguishes several levels of checking. SHOW BACKUP ... WITH check_files verifies that expected backup files are present. A schema-only restore validates the schema without restoring rows. Adding verify_backup_table_data reads the backed-up data, checks its checksums and discards it before writing rows into the destination.

Those operations answer useful questions about the backup that exists. They do not, by themselves, compare every intended source row with the captured result. If a producer omitted a table's row data, checking the integrity of files it did produce is a different test from proving the missing rows were captured. That distinction follows from the documented checks and the release's description of silent omission.

A full restore supplies a destination whose contents can be examined. For this regression, the decisive observation is that the final included table has its expected data while the excluded table remains empty by design. The expected table set must come from the recovery requirements or source state. Inferring it only from the backup would risk accepting the omission as the intended scope.

This does not make lightweight validation unhelpful. File presence and checksum checks can detect other failures with less work than a complete restore. Their results need descriptions that match what they actually checked. A successful file validation is evidence of file availability, while a content comparison after restoration addresses this particular completeness defect.

Schema changes receive separate corrections

Version 26.2.3 also fixes ordinary writes that could fail during certain concurrent schema changes with an error about restricted access to crdb_internal and system. The application statement did not have to reference those namespaces. The affected interval involved adding an expression index, adding a computed column or changing a column type.

That error could therefore misdirect diagnosis toward the application's privileges even when the relevant condition was a schema change in progress. Another fix addresses foreign-key cascades running while columns are added or dropped from a child table. These are concurrency conditions, so exercising only the completed schema would omit the transition that produced the fault.

The backup omission remains the change that most affects my reading of this patch. It connects a valid configuration choice to an invalid loss of unrelated backup content. A recovery test that retains the excluded-table boundary and verifies the final included table gives a specific answer about the corrected behavior, alongside the ordinary file-integrity checks.