April 17, 2023·4 min read·engineeringcockroachdbschemascorrectnessstorage

CockroachDB repairs column-family cleanup during schema rollback

A failed ADD COLUMN could leave primary-index data without matching descriptor metadata. The April advisory identifies concurrent writes and a newly created single-column family as the relevant conditions.

Rolling back a column addition must remove the physical effects of that addition as well as its catalog definition. CockroachDB's declarative schema changer missed that requirement when a new single-column family received concurrent writes and the schema change later failed. The rollback could remove the descriptor metadata while leaving the new family's data in the primary index. The table could then become inaccessible.

The April 17 advisory A-99796 places the affected stable range at 22.2.0 through 22.2.7. It says ALTER TABLE ... ADD COLUMN began using the declarative schema changer in 22.2.0. Release 22.2.8 adds validation to rollback operations to account for the problematic state. The advisory also supplies a way to disable the declarative changer before executing a column addition on an affected release.

I value database behavior that makes schema failure predictable. CockroachDB's online schema machinery has to coordinate live writes with a changing representation, and rollback is part of that behavior. Preserving a usable table after the change fails is as significant as producing the desired schema when it succeeds.

Column families have a physical representation

A column family is not merely a grouping shown in a schema editor. The descriptor definition in CockroachDB 22.2.8 defines a family as columns stored together in one key-value entry. Its descriptor carries a family identifier and the names and identifiers of its columns.

That representation connects table metadata to the entries used to store row data. The descriptor tells the database how those entries correspond to columns. Deleting metadata while leaving an entry that still depends on it creates a disagreement between the physical representation and the schema used to interpret it. The advisory identifies that disagreement as the primary-index corruption in this case.

The same source documents a single-column storage optimization. A family normally stores a tuple containing column identifiers and values, while the single-column form can omit the column-identifier prefix. A default_column_id supplies the identity needed to interpret that value. This is supporting context for the importance of descriptors, rather than a claim that the advisory attributes the entire bug to that optimization.

The relevant point is that stored values and their description must remain consistent. A column-family descriptor is part of interpreting table data, and a schema rollback cannot be complete if the data still requires a family definition that rollback has removed.

Concurrent writes create the state rollback must handle

The advisory gives a specific combination of conditions. ADD COLUMN creates a new family with one column, the command overlaps an UPDATE or INSERT, and the schema change then fails and rolls back. Concurrent writes can place the new family in the primary index before the schema change has reached successful completion.

If that change succeeds, the new metadata remains available to describe the written data. The problem appears on the failure path, when cleanup removes the added column or family from the table descriptor. The primary index can retain the corresponding entry. A rollback intended to restore the earlier schema has therefore left behind data from an intermediate representation.

This explains why a successful isolated ADD COLUMN does not exercise the reported failure. It also explains why the concurrent write alone is insufficient to characterize exposure. The failure and rollback are necessary parts of the advisory's sequence. The source does not claim that every online column addition in the affected versions corrupts its table.

The fix adds rollback validation for this situation. That description should remain as specific as the source. The advisory does not publish a complete step-by-step account of the new internal validation, so there is no basis for claiming an additional storage rewrite or repair pass that it does not name.

The workaround changes the schema execution path

Before upgrading, the advisory permits disabling the declarative schema changer for the session that will run the column addition. The session form is a concrete mitigation for avoiding the affected implementation.

SET use_declarative_schema_changer = 'off';

Applying this in the appropriate session before ALTER TABLE ... ADD COLUMN matters. A session variable in an unrelated connection does not establish the setting used by a migration connection. The advisory also names a cluster-level default for administrators needing a broader change, but the direct session form is sufficient to show which implementation choice the workaround controls.

Disabling the new implementation does not mean abandoning the column addition as an operation. It changes the schema changer used to execute it. That makes the workaround more precise than a general recommendation to avoid all writes during every migration. The advisory offers a specific path around the defective implementation while the cluster remains on the affected release.

My preference is to retain the difference between avoiding the defect and repairing an affected table. Changing the session setting prevents a later operation from using this path. The advisory does not say that it reconstructs metadata or removes inconsistent entries from a table that has already become inaccessible.

The corrected release addresses future rollback handling

The upgrade recommendation is 22.2.8 or later for users on 22.2.0 through 22.2.7. As of the advisory, 22.2.8 is the named maintenance release carrying the fix. That connects the affected stable interval to a definite correction without relying on an unspecified future release.

The report also establishes why data-plane availability and schema-job status need separate interpretation. A failed schema job is expected to report failure, but the additional harm here is leaving the table unusable afterward. Returning the descriptor to an earlier definition does not complete recovery if the primary index retains an incompatible representation.

The substantive requirement is consistency between the table's stored families and the descriptors that survive rollback. Online schema changes permit normal writes to overlap the transition, so their failure handling must account for values written during that transition. The April correction extends rollback handling to the particular single-column-family state described by the advisory.