Reaper 1.4.3 corrects repair coordination and stored state

Reaper was holding a segment lead while sleeping after the repair had finished. That ordering could produce an assertion failure when the runner eventually tried to release the lead. The 1.4.3 release on May 21 includes a small change that moves the pause outside the period of ownership. It also corrects stored repair timestamps and how Cassandra-backed storage executes migrations.

I find Cassandra compelling where an application needs active-active operation at scale. That attraction comes with an interest in the machinery that makes ongoing operation manageable. Reaper is especially interesting here because its own coordination and stored history need to describe the repair work accurately. This release addresses those details without changing the Cassandra server itself.

Release the segment before sleeping

The segment-runner change shows the old and new ordering directly. Previously, the runner acquired the lead, called runRepair(), calculated an intensity-based delay, and slept inside the protected block. A finally block released the lead after that sleep. Holding ownership therefore extended beyond the repair operation into the intentional pause.

The new implementation remembers whether runRepair() returned true. It releases the lead in the existing finally block and then performs the intensity-based delay if repair ran. The pause remains part of the execution path. Moving it does not remove the pacing mechanism or make the configured intensity irrelevant. It changes which part of that execution requires ownership of the segment.

The commit message includes the assertion that motivated the change. CassandraStorage.releaseLead reported that it could not release the lead on a segment, with the call arriving through SegmentRunner.releaseLead and SegmentRunner.run. The patch specifically identifies these as erroneous exceptions. That is narrower evidence than a claim that the release makes every repair complete faster or fixes every leadership failure.

This is a useful concurrency distinction. Ownership protects the repair operation, while the delay controls pacing after that operation. Combining those lifetimes forced a completed operation to retain coordination state during an idle period. Releasing ownership at the end of the protected work makes the code agree more closely with what the runner has actually finished.

A paused run has no completion time

A separate timestamp correction changes which repair-run states should carry an end_time. Earlier migration SQL included PAUSED among states treated as finished. The correction removes PAUSED from that group and adds DELETED. A paused run can therefore retain its pause information without also presenting a completion timestamp.

The new H2 migration expresses part of the correction as follows. This is migration SQL for the Reaper storage database, not CQL to execute against application tables.

UPDATE repair_run
SET end_time = NULL
WHERE end_time IS NOT NULL
AND state NOT IN ('ERROR', 'DONE', 'ABORTED', 'DELETED');

Other statements supply an end time from the start time for terminal records that lack one and remove pause timestamps from records outside the paused state. The PostgreSQL variant uses an explicit timestamp cast when assigning null. These statements repair historical records as well as correcting the state categories used by the migration.

A stored timestamp is part of an operational interface. If a run is paused but carries a completion time, software reading that record has conflicting descriptions of the same work. Reconstructing the exact historical completion time is a separate problem. Using an existing start time as a fallback supplies a consistent record under the migration rules, without establishing when the original operation actually ended.

I care about this distinction because a repair interface is useful only when its state can be interpreted. An operator needs to distinguish work that stopped temporarily from work that reached a terminal state. The timestamp correction makes that distinction explicit in the persisted data, where API consumers can encounter it as well as people using the interface.

Cassandra storage needs the migration code too

The Cassandra migration change deals with migrations that need Java code alongside their CQL scripts. The revised loop advances one version at a time. After running a script, it looks for the corresponding numbered MigrationXXX class and invokes its migrate method. This replaces a sequence that called particular migration classes individually around the script migration process.

That ordering matters for the timestamp repair. The change adds Migration019, which invokes FixRepairRunTimestamps.migrate(session), together with an intentionally empty CQL file for that version. A migration version can therefore represent a data transformation implemented in Java even when it needs no CQL statement in its companion script. The code also corrects the repair-run state decoder to read a string and use RunState.valueOf, replacing an integer-based lookup.

These are changes to the Reaper storage upgrade path. They do not imply a schema change in the Cassandra application keyspaces being repaired. The initialization code also checks for other running Reaper instances before migrating storage at the relevant existing schema versions. That check is a concrete constraint on upgrading a shared Reaper database, independent of the Cassandra server release deployed behind it.

Reaper 1.4.3 consequently affects both active coordination and the interpretation of existing records. The lead-release change shortens the ownership interval to the repair work. The storage changes apply the timestamp correction through the numbered migration mechanism, including the Java transformation that an empty CQL placeholder cannot perform on its own.