Reaper 4.2.5 lets completed repairs stop blocking schedules

Reaper 4.2.5, released on June 11, fixes scheduled repairs that fail to run again after an earlier run completes. The release contains a small change with an operationally significant consequence. A schedule could remain present while its completed history prevented it from starting the next run.

I regard repair orchestration as part of the cost of choosing Cassandra. The database interests me for active-active operation at scale, and I would choose it only when the requirement warrants that machinery. An orchestrator's ability to complete one repair is therefore only part of the evidence I would want. Its ability to repeat the work on the configured cadence matters separately.

The fix in pull request 1686 explains how the regression arose. A previous change reduced a helper to a comparison of the run's cause string with the schedule's cause string. The scheduler then used that ownership check as if it also proved that the run should postpone a new activation.

Historical ownership does not establish active work

The cause string records an association. In the tagged SchedulingManager implementation, a scheduled run carries a cause containing the schedule identifier. Comparing that value with the current schedule can establish where the run came from.

A completed run retains that association. Its history still belongs to the schedule, even though the work has finished. The patch description says terminated runs in DONE, ERROR or ABORTED states could therefore continue to block the schedule. The blockage lasted until storage purged the previous run.

That created an unintended relationship between history retention and scheduling. The patch author describes schedules effectively running on the purge interval instead of their configured cadence. I would not respond to that mechanism by shortening retention. Doing so could remove the blocking record, but it would leave the scheduling decision dependent on the same incorrect predicate.

Keeping old run records is useful for explaining what happened. Their presence should not, by itself, mean that work remains in progress. The fix restores a state check at the place where the scheduler decides whether another activation must wait.

The final predicate checks state and association

The patch introduces a separate helper for deciding whether a run blocks a schedule. The final implementation requires the run to be active or not yet started. It also requires that the run came from the same schedule. Both conditions must hold.

The exact expression is short enough to inspect directly. This excerpt is the predicate in the released implementation.

return (repairRun.getRunState().isActive()
        || RepairRun.RunState.NOT_STARTED == repairRun.getRunState())
    && repairRunComesFromSchedule(repairRun, schedule);

The review history is useful because the first patch version had a broader condition. It allowed any active run on the repair unit to block the schedule. The second revision requires schedule association for the active case as well. A manual run or a run from another schedule no longer blocks this activation through that predicate.

That last qualification matters. I would not turn it into a general claim that all possible overlapping repairs are allowed or harmless. This helper answers a particular scheduling question. Other execution constraints need their own evidence. The useful fact here is that this postponement decision now depends on the state and ownership of the run being examined.

The original cause-only helper remains available for completion registration. That preserves the distinction between identifying a schedule's historical work and deciding whether unfinished work blocks its next activation. A single helper no longer has to represent both meanings.

The regression tests exercise the next activation

The tagged scheduling tests include a due schedule with a previous run in DONE state. The test expects the scheduling manager to start another run. This reproduces the important lifecycle boundary without waiting for a real scheduling interval.

A companion case leaves a previous run in RUNNING state and expects postponement. Another case supplies an active run with a different cause and expects the schedule to start. Together, those cases make the intended predicate visible through behavior.

I like this form of regression coverage because a completed first run would pass an incomplete operational check. The defect becomes observable when the schedule attempts its next activation while the earlier record still exists. A test that purges the record first would remove the condition that needs coverage.

For a deployment acceptance check, I would retain the earlier completed history and observe a subsequent due activation. I would also verify that the same schedule's unfinished run still postpones a new start. Those observations would test the behavior this patch changes without needing to infer it from a successful process startup.

I would keep the test tied to a specific schedule identifier. A repair started manually could show that the execution machinery works while leaving the recurring scheduler untested. Recording the identifier and the retained previous run would make the acceptance result easier for another operator to interpret.

Monitoring should follow recurring progress

The earlier 4.2.0 release added an unfulfilled repair schedule metric. Its change proposal explains that the metric accounts for the configured interval and overdue activation. That is relevant context for a defect in recurring execution.

I would use scheduling evidence to investigate a schedule that stops progressing even though Reaper remains available. The configuration and the existence of an old successful run would not establish that the next interval completed. A useful investigation would identify the schedule's next activation and the state of the run records associated with it.

After adopting this patch, I would examine any schedule already delayed by the defect. A new executable contains the corrected predicate, but an operator still needs to establish what work is overdue in the particular installation. I would preserve the run history while checking that the next intended activation can proceed and that its eventual completion appears in the schedule's monitoring.