Reaper 2.0.3 fixes the storage cost of sidecar metrics

A repair service can create a storage problem through the metrics it collects to supervise the database. Reaper 2.0.3 addresses that problem in its sidecar mode by replacing the metrics table and changing how observations occupy rows. Its other fix corrects a wait loop that could abort repair segments before their timeout. Both changes concern the reliability of the service responsible for managing repair.

The March 2 release contains just two listed changes. Its short description of the metrics fix understates the amount of data modelling involved. The implementation commit introduces node_metrics_v3, changes its partition key and rounds stored timestamps. These details explain what the release actually reduces and what kind of history the resulting table retains.

I am interested in Cassandra because active-active operation at scale is a difficult problem it is designed to address. Repair automation belongs in that assessment. A service that inspects and coordinates the database also needs bounded storage and an accurate account of whether work has finished.

A metrics table with a different purpose

The new table partitions on cluster, metric domain, metric type, time bucket and host. Its clustering columns include the timestamp and the names that identify a particular metric. Including the host in the partition key makes a node part of partition selection. A query for one host can therefore address the corresponding partition directly.

The writer also changes the time bucket from an hourly representation to ten-minute intervals. It rounds the minute down to a multiple of ten and clears seconds and milliseconds. That rounded time supplies both the bucket string and the stored timestamp. The timestamp change is consequential because the timestamp participates in the row key.

Successive observations of the same metric within one interval therefore target the same row identity. That conclusion follows from the new writer and schema together. The implementation reduces the number of distinct stored observations as well as narrowing the partitions that contain them. Describing the fix solely as shorter time buckets would miss this second mechanism.

For example, observations of a given host and metric at 10:03 and 10:08 both map to the 10:00 interval. If their other key components match, they do not establish separate timestamped rows in this schema. This example describes the rounding in the source, without assuming anything about the observed metric values. The table is suited to retaining a recent observation for the interval, with the identity of the metric supplying the remaining clustering components.

The migration defines a default time to live of 3,600 seconds and a gc_grace_seconds value of 300. It also drops node_metrics_v2 before creating the replacement. That is a schema migration with an explicit discard of the old metrics table. It should not be described as a transparent preservation of historical samples.

Readers must follow the new partitioning

Changing the writer alone would leave readers asking for the wrong buckets. The same commit adjusts the Cassandra storage reader to advance through time buckets in ten-minute increments. Each metrics query binds the host alongside the other partition-key components. The old prepared statement for reading cluster-wide metrics disappears from this path.

The client-request latency reader also changes its requested lower bound. It previously looked back one minute and now looks back eleven minutes. The code comment explains that it looks in the last two time-based partitions to obtain a result. A reader near the start of a bucket needs to consider the previous bucket if the useful sample was written before the boundary.

This is a useful distinction between retention and retrieval. A table can retain an observation while a query fails to address the partition containing it. The wider lookback follows directly from changing the partition interval. It makes the read path agree with the representation introduced by the writer.

The migration code attempts to configure TimeWindowCompactionStrategy for the new table when the Cassandra versions permit it. It sets a ten-minute compaction window. The schema also supplies a size-tiered compaction definition, so the presence of a migration helper matters when explaining the final configuration. The commit checks the lowest node version before attempting the time-window change.

A timed wait is not proof of timeout

The repair-segment fix addresses a separate interpretation error. The commit message says a segment could abort before its timeout when condition.await waited slightly less than requested. Previously, the loop used elapsed time around the wait to decide whether to break.

The replacement takes the boolean result returned by condition.await and combines it with the segment state recorded in storage. A stored DONE state also ends the wait. The outer loop continues to compare the current time with the overall deadline, and unfinished iterations renew leadership.

Those conditions represent different facts. A wait returning, a segment completing and an overall deadline passing are separate events. The fix stops inferring completion or timeout from the duration of a single wait. It uses the wait result and recorded segment state while keeping the deadline as the limit on the loop.

I value this level of precision in repair tooling because its decisions affect work already running elsewhere. Reaper 2.0.3 makes two such decisions more explicit. Its metrics reader now follows the storage intervals its writer creates, and its segment runner uses the condition result and persisted completion state when deciding whether to stop waiting.