Cassandra 4.1.9 repairs Paxos progress during mixed-version operation
Two Paxos fixes address disagreement over empty and expired state during upgrades. The patches explain why retrying a missing commit could never repair the apparent difference.
A coordinator can keep retrying a repair that cannot change the state it sees. Cassandra 4.1.9, released on 19 May, fixes two mixed-version Paxos cases with that shape. Differences in empty-ballot representation and expired state could make the prepare phase repeat until a request timed out.
Cassandra's availability model is a large part of its appeal to me. Rolling transitions between versions are therefore particularly interesting. They require nodes to agree on the meaning of old state as well as on the messages exchanged during healthy operation.
Empty state had two representations
The CASSANDRA-20493 patch changes ballot decoding. An older representation of the no-ballot value used a different low portion of its UUID. The newer implementation used another representation for the same conceptual absence.
A newer coordinator could interpret that difference as evidence that an older replica missed the most recent commit. It then attempted to send the newer empty value back to the replica. The apparent disagreement concerned how absence was encoded, rather than a useful application mutation waiting to be applied.
The patch recognizes the old no-ballot encoding while reading it and converts it to the current no-ballot value. That normalization happens at the conversion boundary. Callers can then compare the internal representation without rediscovering the version distinction.
The issue report explains why the difference sometimes caused only another retry and sometimes prevented progress. Expired Paxos data still present on disk could shadow the attempted update. The coordinator continued observing the same mismatch after its repair attempt.
That is a useful compatibility distinction. A representational difference might appear harmless in a fresh-state test while interacting badly with retained expired state. The bug requires considering the data lifecycle in addition to the protocol version.
Expired commits created a second loop
The CASSANDRA-20514 patch addresses a related case. Its regression test constructs a newer coordinator with two older replicas. One replica has a recent-commit record that the other has already converted into tombstones.
The coordinator sees the record on one replica and attempts to repair the apparently missing commit on the other. On that replica, the tombstones defeat the attempted live update. A subsequent prepare observes the same difference, so the repair cannot satisfy its own completion condition.
The implementation restores an age check for legacy Paxos state purging. It obtains the legacy Paxos TTL using table metadata and compares it with the most recent commit's ballot time. If the commit is older than that retention window, it returns no replicas requiring this commit-repair step.
The patch comment explains the resulting responsibility. Hints and repair remain responsible for propagating such old committed data. The prepare path stops insisting that expired Paxos metadata be recreated through an update that tombstones can suppress.
This is not a general instruction to ignore missing commits. The check is conditional on legacy purging behavior and the commit's age. The ordinary agreement check remains for state within the relevant retention period.
The test creates the historical state directly
The added mixed-mode test uses three nodes and upgrades the coordinator. It creates a keyspace with two replicas for the selected key, then arranges different Paxos states on those replicas. One gets a tombstoned commit and the other retains commit data.
The test disables automatic compaction so the intended historical records remain available during the reproduction. That is fixture control, rather than advice to disable compaction in a deployed cluster. Without it, cleanup could remove the state needed to trigger the bug.
A serial read then exercises Paxos through the upgraded coordinator. The expected outcome is that the operation does not time out. This makes progress the regression assertion, alongside the constructed mixed-version state.
My preference is for this kind of upgrade test. It preserves the old state that exposes the compatibility problem and then asks the new coordinator to operate against it. Testing only an entirely upgraded cluster would remove the condition the patch is meant to handle.
Other fixes address separate waiting paths
The 4.1.9 changelog also includes a wait-queue correction and virtual-keyspace permission changes. They belong to different code paths and should not be presented as additional causes of the Paxos loop.
CASSANDRA-20084 describes an uninterruptible wait built by retrying an interruptible operation. The interrupted operation cancelled its signal, so retrying it could wait on an object that would no longer receive notification. That is another progress failure, but its mechanism is local synchronization.
For operators, May's Paxos fixes are most relevant during mixed-version operation involving the older state formats described by the patches. They normalize the legacy empty ballot and stop attempting to recreate sufficiently old commit metadata under legacy purging. Both changes let the prepare phase move beyond a disagreement that repeated repair could not resolve.