Reaper 0.7 coordinates repair across multiple instances
Reaper 0.7 uses a shared Cassandra backend to coordinate repair work among several instances. Conditional ownership records and datacenter-aware configuration explain the design.
Running several repair schedulers introduces an ownership problem. Each process can discover unfinished work, but they need an agreement about which process may execute a particular segment. Reaper 0.7 addresses that problem with shared Cassandra state. Its 22 September release allows several Reaper instances to share one Cassandra backend and announces full support for clusters spanning datacenters.
The important change is how the repair service survives the loss of one of its own processes. A database can distribute its data across nodes while an external maintenance service still depends on a single instance. Making the service redundant requires more than starting another copy. Both copies need persistent progress information and a way to avoid independently claiming the same work.
I am drawn to Cassandra when an application needs active-active operation at scale. A repair service that can also run in several places fits that operational requirement. Its usefulness depends on how it coordinates work and accesses nodes, so the release's storage and datacenter settings deserve attention alongside the headline.
Repair progress belongs in shared storage
The 0.7 README describes Reaper as a centralized, stateful tool that runs repairs in segments. It can opportunistically run repairs in parallel on different nodes. The system stores its full state in a database, exposes a REST API, and communicates with registered Cassandra nodes through JMX.
The README also says Reaper does not cache state changes for running repairs and registered clusters internally. Changes in storage therefore become visible to the running system dynamically. This is relevant when several instances share the database because process-local state would otherwise introduce another source of disagreement about repair progress.
Memory storage remains an option, but the README limits its intended use to testing and notes that it is not persistent. The packaged example configuration still selects storageType: memory. Copying that example without changing the storage choice does not create the shared durable backend required by the release's fault-tolerant mode.
This distinction separates two independent deployment choices. Multiple service processes provide more places to execute scheduling work. A common persistent backend provides the state those processes need to cooperate. The release announcement explicitly associates the new multi-instance arrangement with Cassandra storage.
Conditional records assign ownership
The tagged CassandraStorage implementation prepares a conditional insert into the leader table. This insert uses IF NOT EXISTS and records the Reaper instance identifier, host, and heartbeat time. The takeLead method returns success only when Cassandra reports that the conditional statement applied.
If the conditional insert does not apply, the implementation treats that result as another instance having taken ownership. That is a concrete coordination decision recorded in the shared database. An ordinary unconditional write would allow competing instances to replace one another's ownership records without the same test.
Renewal and release also include ownership checks. The update that renews a record requires its reaper_instance_id to match the requesting instance. Deletion uses the same condition. A process cannot use those operations to renew or remove a record that currently identifies another owner.
The fault-tolerance schema migration gives leader a default time to live of 600 seconds. Ownership is consequently time-bounded unless the record receives further writes. That matters when a process disappears without performing a normal release, since a permanent ownership row would leave another recovery problem for the scheduler.
The schema also adds running_reapers and node_metrics, each with a default time to live of 180 seconds. The storage implementation sends an instance heartbeat every minute. These records describe different things, so their expiration intervals should not be conflated. A record of a running service instance and a claim on repair work have separate roles in the design.
I find these ownership conditions more informative than a general claim of high availability. They identify what the shared backend arbitrates and what information expires. The design also makes the backend part of the repair service's coordination dependencies, which matters when deciding where to place that backend.
Datacenter placement changes available evidence
The tagged example configuration documents three datacenterAvailability values. ALL applies when Reaper can reach JMX ports on nodes across every datacenter. LOCAL applies when it can only reach JMX on nodes in its own datacenter. EACH describes a Reaper instance in every datacenter and the need to avoid involving nodes under duress in repairs.
These settings express an actual network-access distinction. A service's ability to contact the Cassandra backend does not establish that it can reach JMX on every node being repaired. The release's multi-datacenter support needs an arrangement that matches the access available to the deployed instances.
The node-metrics table helps explain why placement is relevant to scheduling. Its columns include pending compactions, whether a repair is running, and active anticompactions, together with host and datacenter information. Shared storage can carry observations about nodes as well as ownership decisions. Those observations have an expiration period, which limits how long an old metric remains in that table without refresh.
A configuration excerpt reproduces the explicit all-datacenter choice in the packaged example:
datacenterAvailability: ALL
That value is appropriate only under the JMX reachability condition documented beside it. Choosing a mode based on the number of service processes alone misses the network distinction the setting describes. An instance in each datacenter and an instance able to contact every datacenter are different arrangements.
Reaper 0.7 combines shared repair state with conditional ownership and configurable datacenter access. Operators gain the ability to run several cooperating service instances, while the backend records who owns work and which instances are reporting. The release's configuration makes JMX visibility an explicit part of that deployment, including the case where each datacenter needs its own Reaper instance.