Reaper 5.0.1 keeps repair polling alive during cleanup

A repair can finish while another thread is still collecting its status notifications. In Reaper's HTTP management path, that overlap could leave the poller holding a job whose notification executor had already disappeared. Reaper 5.0.1, released on September 1, fixes the resulting poller-termination problem and includes further dependency vulnerability updates.

I am interested in Cassandra because of the active-active requirements it can address. That interest includes the management software needed to operate it. A failure in status collection matters because it can interrupt the information an orchestrator uses to follow ongoing repairs.

The useful detail in this patch is how a local cleanup event could affect future work. The repair being removed was only one entry in a collection. An exception escaping its processing could affect the recurring task responsible for polling the rest.

Cleanup and polling observe different moments

The cleanup-race change concerns HttpCassandraManagementProxy. The proxy keeps tracked jobs and separate maps for repair status handlers and their executors. Removing a repair requires cleaning up those associated objects.

The original cleanup ordering removed the handler and executor before removing the tracked job. A poller could therefore observe the job while the machinery needed to dispatch its notifications was already gone. Even reordering the removals cannot invalidate an entry a concurrent iteration has already captured.

The released HTTP management implementation addresses both cases. Cleanup removes the job from the tracker first, which stops new iterations from selecting it. Dispatch also checks for absent executors and handlers, which handles entries retained by an iteration already in progress.

Those checks express a lifecycle fact. An absent handler during concurrent cleanup can be expected. Treating that state as an impossible condition makes ordinary removal capable of interrupting unrelated monitoring work.

A recurring task needs an exception boundary

The proxy schedules the notification tracker with scheduleWithFixedDelay. The regression-test description explains the relevant executor contract. An unchecked exception escaping the recurring task suppresses its future executions.

That changes the consequence of a null lookup. The immediate failure concerns one notification, but the recurring poller can then stop running. A database repair may still exist while the component following its status no longer makes progress.

The new tracker wraps processing of each job entry in a RuntimeException handler. A failed job logs a warning and does not prevent the loop from processing the next entry. The warning also states that the entry can be retried on the next poll.

The location of that boundary matters. Catching around the whole loop would still allow one failing entry to prevent later entries from being processed on that tick. Catching per entry narrows the effect to the job that failed. The code retains information about the failing job in its log message.

I value that specificity in an operational tool. A warning tied to a job identifier gives an investigation a starting point, while the rest of the status collector can continue its work. Merely keeping the process running would leave the more important question of polling progress unresolved.

Dispatch tolerates the remaining race window

The implementation obtains the executor and handler before submitting the notification. Missing values cause a logged skip. The submitted task captures the handler reference instead of performing another map lookup later.

There is still a possible transition after the executor lookup. Cleanup can shut down the executor before submit runs. The new code catches runtime failures from submission, including rejection during that interval. A null check alone would not cover that ordering.

The implementation also checks the job response. A null status-change collection or malformed repair identifier produces a warning and returns from processing that entry. An unknown progress-event value is handled during dispatch.

These branches do not all have identical retry behavior. Processing increments the notification count after dispatch returns, including when dispatch has skipped a notification because its handler has disappeared. The per-job exception boundary, by contrast, permits another poll after a processing failure. Calling the whole change a blanket retry mechanism would obscure those distinctions.

The test controls the interleaving

The tagged proxy tests include a test that blocks the poller while it retrieves job status. Cleanup then removes the repair handler before the poller continues.

That reproduces the relevant ordering directly. It does not depend on repeatedly running a fast loop until the scheduler happens to create the race. The test can establish whether the poller lets an exception escape under that controlled interleaving.

The accompanying dependency change is separate maintenance work. The tagged server build configuration selects Netty 4.2.17.Final for the handler and managed codec artifacts. Dependency management also selects updated HTTP components and AMQP client versions.

For installations using the HTTP repair-management path, the cleanup fix changes a concrete failure condition during notification collection. Its value is that removal of one repair can coexist with continued polling. The controlled concurrency test documents the ordering future changes to those maps and executors need to preserve.