Cassandra beta3 coordinates truncation with repair
Cassandra 4.0-beta3 aborts relevant repair sessions during truncation and propagates replica truncation failures. The changes address deletion races and the meaning of a successful response.
Deleting a table's contents on one replica does not finish the operation if another activity can immediately stream the old data back. Cassandra 4.0-beta3 addresses this interaction between truncation and repair. It also changes the response path so that a failed truncation on a replica makes the overall operation fail. These changes concern the meaning of completion in a distributed administrative command.
The Apache release record dates beta3 to November 4, 2020. Its changelog lists CASSANDRA-15854 for aborting repairs when a truncation request arrives and CASSANDRA-16208 for failing requests when truncation fails on a replica. Read together, the entries identify a concurrency problem and an error-reporting problem within the same operation.
I find Cassandra compelling for active-active systems that require its scale. That makes a command affecting every copy of a table more interesting than its short CQL spelling suggests. Its result needs to describe what happened across the participating machines, while concurrent maintenance must respect the deletion.
Why repair can conflict with truncation
The repair issue describes the race directly. Truncation can clear data on one node while ongoing repair streams data from another node that truncation should have deleted. The problem arises because different nodes can reach the truncation at different times. An operation that copies state between them has to account for that interval.
The beta3 table-store implementation aborts parent repair sessions whose table identifiers include the table being truncated. The predicate ties cancellation to the affected table. It does not express a rule that every repair in every keyspace must stop whenever any table is truncated.
This check runs inside the truncation work before the code discards SSTables. The same block notifies the data tracker of truncation and, when automatic snapshots are enabled, creates a snapshot. It then discards the relevant SSTables and truncates indexes and views before recording the truncation position.
The ordering makes the repair interaction explicit at the point where the table contents are being removed. The source comment gives the reason for aborting sessions. Data cleared locally must not return through a repair that is still operating on the earlier state elsewhere.
Truncation also has a recovery boundary
The table-store method documents two goals. It aims to delete data written before invocation while avoiding deletion of data outside the snapshot boundary. The implementation flushes when durable writes or automatic snapshots require it, then determines the truncation time before carrying out the removal work.
It also retains a commit-log position in the system keyspace. The source explains that removing SSTables affects how commit-log segments could be replayed after a restart. Recording the truncation position prevents normal recovery from treating the deleted data as something to restore through replay.
That gives truncation more than one interaction with old data. Repair can copy it from another replica, and recovery can encounter earlier commit-log entries. The implementation handles those through different mechanisms. Aborting relevant repairs addresses concurrent transfer, while saving the truncation record addresses the replay boundary.
These are consequences of reading the operation as a sequence with durable state. A command returning does not eliminate the data paths that exist around it. Each such path needs a defined relationship to the truncation time and table identity.
Replica responses now carry a failure result
The truncation response handler checks the success flag in a response. When a replica reports failure, the handler records the replica address, signals the waiting caller and returns without counting that response as success. The waiting method then throws a TruncateException identifying the failing replica.
Successful responses increment a counter. Once that count reaches the expected number, the handler signals completion. The method waiting for the result also distinguishes a timeout from a reported failure. A timeout says that the expected responses did not arrive within the allotted time, while a failure response supplies evidence that a particular replica could not complete truncation.
This distinction improves what the client can learn from the operation. Receiving messages from replicas is not sufficient evidence that each replica completed the requested change. The response payload and the expected response count both participate in deciding whether the command succeeded.
The replica verb handler provides the corresponding failure response when truncation throws. It logs the error and sends a response whose success field is false. The coordinator-side change therefore has a concrete result to interpret.
I prefer this explicit treatment of failure because a maintenance command needs an assessable outcome. Beta3 checks the reported success of truncation separately from the arrival of replies. It also aborts repair sessions involving the affected table before discarding its SSTables, so the deletion has a defined relationship to concurrent repair work.