Cassandra reverts a schema-disagreement fix to prevent corruption

A change intended to let Cassandra tolerate temporary schema disagreement could place invalid column metadata into stored data. The August upgrade advisory explains why the project has reverted that change. The replacement releases deliberately retain an older transient failure because the attempted correction introduced a more serious corruption risk.

The August 18 advisory identifies 3.0.23, 3.0.24, 3.11.9 and 3.11.10 as affected. It recommends immediate upgrades to 3.0.25 or 3.11.11 for users on those versions, with schema changes stopped until the upgrade is complete. Users coming from 3.0.22 or 3.11.8 should skip the affected versions. The advisory says some partitions can suffer unrecoverable corruption, revising an earlier understanding that the corruption was recoverable.

I find Cassandra's distributed architecture attractive for applications that need active-active behavior at scale. This failure is a reason to be precise about what the implementation does during disagreement. Temporary differences in schema knowledge can affect serialization, and a representation created to tolerate that difference must not become incompatible persistent data.

The original failure occurred while schemas differed

The earlier report, CASSANDRA-15899, describes queries breaking after a column is dropped but before the schema change reaches every replica. Its example uses a table with a primary key and several integer columns. Dropping one regular column leaves some nodes with a different column definition from the node that has already applied the change.

The reported exception comes from column-subset serialization. A serializer tries to describe a set of columns against a different expected set and raises an IllegalStateException. The ticket says the failure occurs both with SELECT * and with a query selecting only the key and another column. Avoiding the dropped column in the visible projection therefore did not eliminate the reported failure.

That last detail is useful because it identifies a limitation in an apparently plausible application workaround. A query's projected columns do not necessarily describe every column set involved in the internal operation. The report provides a concrete case where changing the visible projection still reaches the same serialization failure while schema information differs.

The problem was transient in the sense described by the advisory. It concerned the interval before schema propagation completed. A failure during that interval still matters to clients, but the scope differs from an invalid representation surviving into files. The later advisory explicitly ranks the corruption risk as more serious than this original issue.

Placeholders represented columns a node did not know

The corrective CASSANDRA-16735 report identifies column placeholders introduced by the earlier patch. Its description explains that read repair could place those placeholders in a memtable. A later flush could then persist them in an SSTable on disk. In some cases, particularly when the actual column was a collection, the placeholder conflicted with the real definition and caused CorruptSSTableException failures.

The important transition is the change in the placeholder's lifetime. A representation introduced to handle incomplete schema knowledge during an operation could become part of data that another operation retained. Once it reached a memtable and then a flush, the problem no longer depended solely on the duration of the initial disagreement.

The report's description does not claim that every schema change corrupts every partition. It names a possible path and a specific kind of conflict. The advisory supplies the operational response because the consequence can be unrecoverable corruption. Preserving both statements avoids understating the consequence or overstating how universally it occurs.

This also explains why ordinary schema agreement after the event is insufficient evidence of recovery. The reported failure path includes persistence of an incompatible representation. Later agreement about the real schema does not by itself establish that no earlier placeholder reached a file. The advisory's recommendation is to leave the affected releases, not to treat eventual agreement as proof that the risk has passed.

The patch removes placeholder construction

The corrective commit makes the implementation change visible. It removes the ColumnDefinition.Placeholder class and the helper that created placeholder column definitions with EmptyType. The column deserializer again throws an error for an unknown column when it cannot obtain the necessary definition from metadata.

That is a deliberate refusal to continue with an invented definition. The removed code had attempted to represent a column for which the node had not received a schema update. The restored behavior can fail the operation during disagreement, which is consistent with the advisory's warning that the older transient issue remains.

The same commit changes the handling of column sets in row serialization and updates the callers of column deserialization. Those edits show why the correction spans several files. A column definition participates in both the description of an operation and the serialization of its data. Removing the special representation requires the surrounding code to stop relying on its behavior.

The patch does not support a claim that existing corrupted SSTables are automatically reconstructed. It changes the code responsible for creating and handling the problematic state. The advisory recommends an upgrade because that removes exposure to this defect, while separately explaining that the older schema-disagreement problem is restored. Prevention of further exposure and recovery of previously corrupted data are different claims.

The regression tests preserve the failure sequence

The corrective commit adds distributed schema tests that exercise the interaction between disagreement and persistence. In one test, a two-node cluster begins with a shared table. A collection column is added internally on one node, leaving the other node without that schema change. The test writes data on the updated node and performs a query while the schemas differ.

It then flushes the second node, applies the collection schema there, restarts it and forces compaction. Another test includes writing a real collection value and additional flush and compaction operations. These steps are significant because the reported corruption involved a placeholder surviving beyond the read into stored data. A test ending at the query result would omit the later operations that expose the incompatibility.

The helper used for the disagreement query accepts a failure only when the exception chain contains the expected unknown-column deserialization error. That makes the test's tolerated failure specific. It does not treat any exception as evidence that the database has behaved correctly. The restored transient error is part of the scenario the patch intentionally permits.

Other changes in the commit update distributed read and write tests to expect unknown-column failures during schema disagreement. These are compatibility changes to the test expectations as well as implementation changes. The test suite records the narrower guarantee restored by the revert, including the cases where an operation is expected to fail.

I value that connection between the diagnosis and the regression tests. The tests continue through flush, restart and compaction because the defect crosses those stages. They represent a concrete causal account of the failure instead of relying only on the absence of a query-time exception.

The supported response accepts a known limitation

The advisory's version guidance follows directly from the history of the patches. The affected releases contain the change for CASSANDRA-15899. Releases 3.0.25 and 3.11.11 revert it through CASSANDRA-16735. Those replacement versions remove this unrecoverable-corruption exposure while remaining susceptible to the older transient issue. Describing them as eliminating every schema-change problem would contradict the advisory.

The recommendation to stop schema changes applies until the upgrade is complete for clusters already running an affected version. That instruction concerns continued exposure during the transition. It does not require an unrelated major-version migration as the remedy. The project identifies replacement releases within both affected maintained lines.

There is also a separate schema-operation warning in the 3.11.11 upgrade notes. They classify ALTER ... DROP COMPACT STORAGE as experimental and advise against production use, with a flag to prevent it. That warning concerns a different operation and should remain distinct from the column-placeholder corruption path.

For a 3.x operator, the August advisory therefore provides a specific version decision and a specific restriction during the upgrade. The replacement code restores rejection of unknown column definitions during disagreement. Its tests retain the later flush and compaction stages needed to check that temporary schema differences do not produce the placeholder conflict described in the report.