July 25, 2018·4 min read·engineeringcassandrarepairreaper

Reaper 1.2.2 distinguishes full and incremental repair units

Reaper 1.2.2 corrects repair-unit lookup and ownership. It also puts background executors under the application lifecycle so repair management shuts down more predictably.

Two repair definitions can name the same cluster and keyspace while requesting different work. Reaper 1.2.2 fixes a lookup that failed to preserve that distinction when the difference was the incremental-repair flag. The reported failure involved adding a scheduled full repair and a scheduled incremental repair for the same keyspace. Treating those requests as interchangeable caused a consistency check in schedule creation to fail.

The July 25 release notes identify the repair-unit matching fix alongside changes to executor management and repair notifications. The common concern is how Reaper manages its own state while directing Cassandra maintenance. A repair scheduler has to retain the meaning of a requested repair across storage lookups, scheduling decisions and process shutdown.

I find Cassandra's active-active capabilities compelling when the problem warrants the operational commitment. That makes repair tooling worth understanding in its own right. Automating maintenance is useful only if the stored definition continues to mean the work the operator requested.

The incremental flag is part of identity

The repair-unit patch names the failing integration scenario and identifies the memory and PostgreSQL implementations as broken in that case. The PostgreSQL lookup previously compared the cluster, keyspace and selected tables, together with other repair restrictions. It omitted the incremental-repair parameter.

The correction adds that parameter to the Java storage call and to the SQL predicate. It is a small change with a direct semantic effect. Two units that differ on incremental repair no longer qualify as the same unit merely because the other selection fields agree.

This matters because retrieving an existing unit is a reuse decision. The scheduler does not create every object independently. It asks storage whether an appropriate unit already exists, then uses the returned object's identifier. If lookup equality is weaker than the requested repair definition, reuse can return an object with a different meaning.

Schedule creation already contains a precondition comparing the returned unit's incremental setting with the request. The patch improves its failure message as well as the lookup. That check helps explain the reported symptom. The scheduler can detect the mismatch after lookup, but detection at that point still prevents the requested schedule from being created.

The storage interface therefore needs the same notion of identity as the code consuming it. A boolean can be as important as the keyspace name when it determines the requested operation. Its size in the serialized record says nothing about its significance for matching.

Reuse requires a consistent deletion policy

The same patch changes when Reaper deletes repair units. In PostgreSQL storage, deleting an individual repair run or schedule no longer attempts to delete the associated unit. Cluster deletion instead removes the cluster's related units. The Cassandra and memory storage paths also receive cleanup logic associated with deleting the cluster.

That adjustment follows from sharing units. A unit referenced by one run can also matter to another run or schedule. Deleting the unit as an incidental consequence of deleting a single reference gives the unit a lifetime shorter than its use. Defining cleanup at cluster removal gives the code an explicit ownership boundary for those objects.

The patch adds checks around cluster cleanup for remaining schedules and runs. Those checks describe an intended ordering of operations. Cleanup of repair definitions follows removal of the work that refers to them. The implementation is easier to reason about when reuse and deletion agree on that relationship.

Schedule creation also moves conflict checking ahead of obtaining or creating a repair unit. A request rejected because of an existing schedule should not need to create an unused unit first. The commit describes tighter constraints on reusing existing units and creating only units that will be used. This changes the sequence in which persistent state appears during a request.

Background threads need an application lifetime

A separate executor lifecycle patch addresses exceptions during shutdown or changes between Reaper instances. It replaces several independently constructed executor services with services created through Dropwizard's lifecycle facilities. The application passes those services into the components that use them.

The distinction is concrete in the cluster resource. Its static executor becomes an instance field supplied to the constructor. Snapshot and repair managers similarly receive application-managed executors. The application scheduler also moves into lifecycle-managed construction.

A thread pool created in a singleton can outlive the work that originally justified it unless something explicitly stops it. Giving the application framework ownership lets shutdown management follow the application's own lifecycle. Passing the executor into the component also makes the ownership visible at construction.

I prefer this explicit ownership for maintenance services. It makes it easier to identify which application instance owns scheduled activity, particularly when the service has both recurring work and requests arriving through its API. The release notes do not attach a performance claim to this refactoring, and its stated purpose is shutdown behavior.

The release also ignores late start notifications in SegmentRunner and synchronizes the repair manager's private start operation. These changes concern the ordering of state transitions around work already in progress. Together with repair-unit matching, they make 1.2.2 relevant to installations whose difficulty is managing repair definitions and lifecycle events, even when no new repair command is needed.