July 6, 2020·4 min read·engineeringcockroachdbschema-changesmaintenance

CockroachDB repairs stale drop jobs after TRUNCATE

A copied drop-job identifier could prevent a truncated table from being renamed. The repair separates preventing new bad metadata from handling descriptors already affected by the bug.

Removing a table's rows can leave a problem that appears only when the table is renamed later. CockroachDB's July 6 advisory A-50587 describes that sequence in the 19.1 and 19.2 branches. TRUNCATE gave the replacement table a drop-job identifier belonging to the old table. A subsequent rename encountered that stale association and could fail to finish.

The affected maintained releases extend through 19.1.10 and 19.2.8. The advisory names fixes in 19.1.11 and 19.2.9 and recommends upgrading. For installations still affected, it recommends delaying renames of previously truncated tables until the update. This is a dependency on an earlier operation, so the table's present name alone cannot identify whether the problematic metadata exists.

I value CockroachDB's explicit consistency model, and schema maintenance needs the same care about state transitions. A table that remains publicly available should not carry metadata identifying it as the subject of another table's pending deletion. The bug is interesting because the initial TRUNCATE can appear complete while an unrelated-looking operation later encounters the inherited state.

TRUNCATE replaces the table descriptor

The 19.2 TRUNCATE reference available at disclosure describes the operation as removing all rows by dropping the table and recreating it under the same name. It requires DROP privilege. Foreign-key dependencies affect whether it can proceed, with RESTRICT as the default and CASCADE extending the operation to dependent tables.

That implementation is different from individually deleting every row while retaining the same table identity. The logical result is an empty table, but the underlying descriptor changes. Cleanup of the old table's stored data can occur asynchronously. A later operation must therefore refer to the new table's metadata rather than accidentally inherit cleanup responsibilities associated with the old one.

The public documentation also classifies TRUNCATE as a schema change with the transactional limitations of that branch. This is relevant when comparing it with a DELETE without a predicate. Both can remove the table's rows, but they do not perform identical internal work or carry identical schema-operation behaviour. A maintenance procedure needs the documented semantics of the operation it selects.

Copying metadata carried the wrong job forward

Andrew Werner's June 23 issue report explains the assignment order. The old descriptor received its DropJobID before the code copied it to construct the new descriptor. The copied metadata consequently included a nonzero drop-job identifier on a table that was not being dropped.

The report describes failures after cleanup of the old table progressed. A rename could try to update a completed job or refer to a job record that no longer existed. This explains why the error might mention job progress even though the user's immediate command was an ordinary table rename. The failing reference came from the table's inherited metadata.

The important distinction is between copying a descriptor and copying user rows. TRUNCATE's replacement is empty. The mistaken inheritance is an internal field from the old descriptor. Understanding that difference avoids interpreting the advisory as a claim that truncation restores the deleted data into the new table.

Prevention changes one assignment's timing

The first repair moves assignment of DropJobID until after creation of the replacement descriptor. Only the old table receives the identifier when its drop begins. The replacement descriptor therefore avoids acquiring a cleanup job that belongs to another object.

Its regression test creates a table, inserts a row and truncates it. The test verifies that the descriptor identifier changes and that the new descriptor has no drop-job identifier. Those assertions express the intended state after the operation. They do more than show that TRUNCATE returned successfully, because the later rename bug depends on the descriptor's contents.

That patch's message explicitly says it does not yet make clusters robust to descriptors already containing the bad field. This is a useful separation in a maintenance fix. Correcting the code that creates state prevents new occurrences. Existing catalogs still require a path that tolerates or repairs the state produced by the earlier implementation.

Existing tables need a repair path too

The follow-up patch for 19.2 addresses those existing descriptors during name draining. When a table is PUBLIC but has a drop-job identifier, it clears the identifier. It also prevents failure to update a drop job from blocking rename progress, logging that failure instead.

The added regression test constructs a descriptor with a bogus drop-job identifier and then renames the table. That models the durable state left by the old implementation. A test that only exercised newly corrected truncations would not cover the same condition, because those truncations no longer create the invalid metadata. The two tests consequently establish different properties about creation and subsequent use of table descriptors.

I regard the two-part repair as the important operational detail. A maintenance release has to work for a database with history, including objects created by earlier defects. The follow-up makes that requirement concrete by presenting the renamed table with metadata that the new truncation path would never generate.

Rename has a distributed completion step

The 19.2 rename documentation describes namespace updates, descriptor propagation and eventual release of the old name for reuse. It warns that renames in this branch are not fully transactional. A successful-looking local namespace change therefore does not describe every step involved in completing a rename across the cluster.

The 19.2.9 release notes, dated July 6, include both TRUNCATE fixes. For an affected installation, the relevant outcome is that future truncations avoid the stale job association and previously truncated public tables can progress through rename cleanup. The release changes the metadata transition and its recovery path together, which is what the table's later operations need.