February 1, 2021·4 min read·engineeringcassandraconsistencymaintenance

Cassandra 3.11.10 repairs SERIAL read correctness

Cassandra 3.11.10 fixes a linearizability defect in SERIAL reads and conditional writes that do not apply. Its upgrade notes explain the performance cost and a separate SASI memory-setting correction.

A database maintenance release can change the cost of a consistency guarantee. Cassandra 3.11.10 does that through a correction for SERIAL reads and lightweight transaction writes that do not apply. The release notes warn that users who make heavy use of SERIAL reads may see a noticeable performance impact, including increased timeouts. The change matters because the previous behavior was incorrect. Keeping the earlier performance characteristics can mean keeping the defect.

The Apache release record dates 3.11.10 to February 1. Its tagged upgrade notes put the consistency correction first in the version-specific section. That placement gives existing users a more useful starting point than treating the update as a collection of interchangeable bug fixes.

I am interested in Cassandra for the difficult distributed systems problems it addresses, especially active-active operation at scale. That interest comes with a preference for explicit guarantees and costs. This release documents a case where correcting behavior has a performance consequence. The warning belongs in an assessment of the upgrade because it describes the price of obtaining the intended semantics.

The compatibility switch restores incorrect behavior

The upgrade notes identify CASSANDRA-12126 as a correctness issue affecting SERIAL reads and lightweight transaction writes that do not apply. They name both SERIAL and LOCAL_SERIAL as read consistency levels where the performance impact can appear. That scope is more precise than saying the release makes every Cassandra read slower. The warning concerns particular operations and consistency choices.

The project supplies an opt-in JVM system property to disable the correction. The notes describe its effect in unusually direct terms. It reverts SERIAL reads to the incorrect behavior of previous versions, and users enable it at their own risk. The property is named cassandra.unsafe.disable-serial-reads-linearizability. Its name makes the lost guarantee visible in a process configuration, but the setting cannot reconcile the old behavior with the corrected semantics.

The distinction is consequential when interpreting an upgrade-related increase in timeouts. The release notes already identify such an increase as a possible effect of the fix for heavy users of SERIAL reads. Restoring the old behavior through the property may change that symptom, but it also changes the correctness being requested from the database. A timeout count alone cannot establish that the opt-out preserves the application's requirements.

The 3.11.10 changelog calls the change a repair to serial-read and non-applying-CAS linearizability. That description also prevents a narrower misreading in which only successful conditional writes matter. The release explicitly includes the path where a conditional write does not apply. An application that examines an unsuccessful conditional result still depends on database behavior covered by this correction.

A memory limit depends on its unit

The same upgrade notes describe a separate failure involving SASI's max_compaction_flush_memory_in_mb setting. Earlier code interpreted this value as bytes. Starting with 3.11.8, Cassandra correctly interprets it as megabytes. A retained numeric configuration can therefore acquire a very different effective size after that change.

Before 3.11.10, the notes say these earlier configurations can cause nodes to run out of memory during compaction. Version 3.11.10 detects the incorrect configuration, logs it and reverts the setting to the default of 1 GB. The changelog identifies the detection case as settings over 100 GB. This is a compatibility repair for configurations whose original numeric value depended on the mistaken interpretation.

The fallback does not rewrite the index into the configuration its owner intended. The notes leave that correction to the user through dropping and recreating the index. That separates two effects of the update. Cassandra can reject an implausible inherited limit and choose a default, while the deployment still needs an explicit decision about the appropriate index setting.

I find this a useful example of why a configuration field's apparent clarity is insufficient. The name already contained its unit. Its implementation disagreed with that name, so correcting the implementation also exposed existing configurations to a change in meaning. The release documents both the defensive fallback and the remaining work needed to correct the index.

Maintenance also changes resource control

The changelog contains other operational corrections with narrower descriptions. Validation compactions now use compaction_throughput_mb_per_sec for rate limiting. Another change prevents an unbounded number of pending flushing tasks. These entries concern control over background work and queued work, respectively. They do not establish a universal throughput improvement, and the release record supplies no benchmark that would support one.

Startup validation also becomes stricter for virtual-node configurations that specify initial_token. The upgrade notes require num_tokens to be present and to match the number of specified tokens. Otherwise the node does not start. This makes a configuration mismatch an explicit startup failure. The behavior is part of the maintenance release's compatibility contract even though the database cannot begin serving requests under the invalid configuration.

For an existing 3.11 deployment, these documented changes identify different consequences of the same update. SERIAL operations receive a correctness repair with a potential latency cost. SASI receives protection against a misinterpreted memory limit. A token-count mismatch can prevent startup. The release's own notes provide the specific conditions under which each consequence applies, including the index recreation required after the SASI fallback.