Cassandra 3.11 starts maintenance with a schema protocol change

Cassandra's move away from tick-tock gives the 3.x feature set a branch intended to receive continuing corrections. It also exposes an important distinction in database maintenance. A bug fix can require changing how nodes communicate, even when applications gain no new query operation. Cassandra 3.11.0 contains such a change because concurrent reads and column additions or removals could corrupt data.

The Apache release record dates 3.11.0 to 23 June. Its upgrade notes describe the protocol transition and the choices available during a rolling upgrade. This is a useful way to understand the new branch. Maintenance means preserving and correcting a chosen implementation, including the distributed compatibility work that some corrections require.

I find Cassandra compelling when active-active operation at scale is an actual requirement. That preference also makes the maintenance path relevant to the choice of database. A supported sequence of corrections helps operators retain the capabilities they need without following every new feature, but each correction still needs an operational explanation.

The branch follows the last tick-tock feature release

Michael Shuler's 3.10 announcement in February explicitly calls 3.10 the last tick-tock feature release and says that 3.11.0 will continue bug fixes on the cassandra-3.11 branch. His June announcement identifies 3.11.0 as the first release of that new series.

That establishes the maintenance intention without predicting how long the branch will remain in service. The relevant change is already concrete. Users of the 3.x capabilities have a named branch on which corrections continue. A feature-release schedule and a bug-fix branch answer different questions, and the announcements make that distinction explicit.

The 3.11.0 changelog shows the resulting work. It includes corrections to duration validation, SASI paging, and schema handling, as well as changes merged from the 3.0 maintenance line. This is not a clean separation between new code and old code. Branch maintenance requires bringing relevant fixes into the implementation that users have chosen.

The inherited feature set still matters when deciding to use this branch. The upgrade notes place GROUP BY, persisted prepared statements, and the slow-query log under 3.10. Reading those headings prevents attributing all capabilities in the distribution to 3.11.0 itself. This release maintains a feature line that already exists.

Concurrent schema changes require a common column view

CASSANDRA-13004 concerns corruption while adding or removing a column. The original report describes an ALTER TABLE adding a bigint column, followed by a row that clients could no longer decode correctly. The reported symptoms included a UTF-8 decoding failure and an implausible timestamp value. Those symptoms show why a schema race can resemble damaged application data.

The release changelog describes the correction as ensuring a consistent view of partition columns between coordinator and replica in ColumnFilter. The upgrade notes state that adding or dropping columns concurrently with a read might cause corruption, and that fixing the bug required a messaging protocol version bump. By default, 3.11 uses messaging version 3014.

This is an internode messaging concern. Nodes coordinating a distributed operation need to agree about the column information used in that operation. A fix that changes the relevant message representation introduces a compatibility condition while a cluster contains nodes running different releases. Installing a corrected binary on one node does not immediately give every participant the corrected protocol behavior.

Schema migration adds another constraint because it relies on an exact messaging protocol version match. The notes therefore distinguish an upgrade that can suspend schema migrations from one that needs migrations during the transition. The appropriate procedure depends on that operational requirement, rather than on a single flag that every installation should copy.

Compatibility mode has a defined purpose

If the cluster can avoid schema migrations during the upgrade, the release notes say to start 3.11 normally. There is no requirement to force the older protocol in that case. Nodes can use the release's default messaging behavior while the upgrade proceeds under the stated schema restriction.

If schema changes are needed during the upgrade, the notes provide a temporary JVM property:

-Dcassandra.force_3_0_protocol_version=true

Starting nodes with that property forces a backward-compatible protocol. Once the whole cluster has upgraded to 3.11, the documented procedure calls for a rolling restart without the property. Compatibility mode is therefore an intermediate state with a second transition needed to reach the normal 3.11 configuration.

While that flag is true on 3.11, the notes explicitly say to avoid adding or removing columns from existing tables. The flag makes a particular upgrade sequence possible, but it does not remove the original risk associated with those operations in compatibility mode. A procedure that enables it and then treats every schema operation as unrestricted would omit a central condition of the documented workaround.

The notes also identify an exception for users upgrading from 3.0.14 or later in the 3.0.x branch. Those users do not need the flag to ensure schema migrations during the upgrade. The source release matters, even when the destination version is identical. An upgrade procedure written only around the destination number loses that distinction.

I regard this as a useful example of what a database maintenance branch must communicate. A promise to fix defects is incomplete without the conditions under which nodes can use those fixes together. Here, the notes describe both the compatibility mechanism and the point at which operators should remove it.

Request timeouts describe more of the operation

Moving onto 3.11 can also expose changes inherited from 3.10. One operationally significant example is the meaning of request timeouts in cassandra.yaml. The upgrade notes explain that these timeouts now apply to the full request time on the coordinator. Previously, they covered the interval between sending a message to a replica and receiving its response.

The earlier behavior also reset the timeout for read repair, a second read to fix a short read, and subrange reads during range scans or secondary-index queries. In 3.10 and later, those subqueries no longer reset the clock. The entire request must complete within the configured timeout.

This changes the interpretation of an existing value without changing its spelling. An operation containing several internal reads no longer receives a fresh timeout budget for each described subquery. The notes say that timeout settings may need adjustment to account for the change. That is a specific semantic change, beyond a general recommendation to review configuration.

The result is a more comprehensive bound on the coordinator's request. Applications still need to interpret timeout responses in their own context, but the database configuration now describes a larger portion of the work than it did under the earlier behavior. A maintenance-branch upgrade can therefore change observed failures even if the application's CQL statements remain identical.

Persisted prepared statements reduce restart work

Another inherited 3.10 change stores prepared statements in the prepared_statements table in the system keyspace. On startup, Cassandra uses that table to preload previously prepared statements. The notes describe the practical result cautiously. In many cases, clients no longer need to prepare statements again against a restarted node.

The mechanism concerns server state across a restart. Persisting the statements gives the node information it previously had to receive again from clients. It does not imply that client preparation logic can disappear, because the release notes do not promise that every possible reprepare becomes unnecessary. The stated benefit is reduced repeated preparation in the cases covered by the persisted state.

For a branch intended to receive continuing corrections, restart behavior is part of normal operation. Rolling out maintenance releases repeatedly exercises node shutdown and startup. A change that preloads prepared statements can reduce one source of client work during that process, while the schema protocol rules determine a separate aspect of compatibility among the running nodes.

The first 3.11 release combines those inherited capabilities with corrections to the implementation. Its maintenance role does not make configuration semantics or messaging compatibility disappear. The documented path distinguishes the source branch, the need for schema migrations during rollout, and the temporary use of the older protocol. Completing that path includes removing compatibility mode when the upgraded cluster can use the corrected default protocol.