Harry reconstructs Cassandra test histories from compact descriptors

Generating a large database workload is easier than explaining exactly what every read should return. A test can complete without an exception while still returning the wrong data. Harry addresses that verification problem by constructing reproducible operations and representing generated values with compact descriptors. The expected database state can then be reconstructed without retaining a full history of every value sent to Cassandra.

Alex Petrov's November 18 introduction explains the design. Harry can generate data for varying schemas, execute modifications, track their progress and verify query results. The article's detailed account concerns a quiescent checker, which validates when no operations remain in flight and the status of every relevant operation is known.

I find that emphasis useful for Cassandra because distributed behavior needs more evidence than successful request completion. My interest in the database comes from the difficult storage requirements it can address. A tool that verifies the result of generated histories helps examine the implementation behind those capabilities without making every test author manually calculate large result sets.

A failing example can become a family of tests

The article uses CASSANDRA-16453 as a concrete example. The issue concerns reconciliation of a range deletion and a partition deletion with the same timestamp. Its reproduction creates disagreement about deletion information across nodes, then queries a clustering range. The reported failure includes a null-pointer exception in the range-tombstone handling path.

For a regression test, the relevant pattern can be expressed independently of many literal values. A table needs a clustering column, the history needs the two deletion forms, and their timestamps need to tie. Harry can vary schema details and operation order around those requirements. That extends coverage beyond one hardcoded table and one particular set of keys.

The benefit is still bounded by the generated test. Varying inputs does not establish correctness for every possible operation. It creates more instances of a specified scenario and checks the results of those instances. The article's example demonstrates how a known bug pattern can supply constraints for a broader test family.

Operation identities reproduce the generated work

Harry gives each operation an identity based on a logical timestamp and a modification identifier. The latter allows multiple identifiable operations at one logical timestamp. Given its identity and the run's configuration, Harry can reproduce the operation instead of retrieving its entire original statement from a retained log.

Different operation kinds need different descriptors. A partition deletion needs a partition descriptor. A range deletion also needs clustering descriptors for its bounds. An insertion or update needs descriptors for the key and the values of affected columns. The generated statement is the concrete expression of that smaller internal representation.

The article describes partition keys and clustering keys as single long descriptors, even when the real keys have multiple components. Each regular or static column value has its own descriptor. This separates the amount of data the database handles from the size of the identity the checker uses to reason about it.

Operation selection is deterministic too. The configuration can assign weights to inserts, updates and deletion forms. A seeded pseudorandom sequence chooses operations according to those weights. Reproduction therefore requires the configuration and seed as well as an operation's position. Recording only a random seed while changing the schema or distribution would not preserve the same history.

The generator also needs addressable streams. The introduction describes obtaining an element from its sequence number and stream identifier, as well as moving to a preceding or following element. Those operations let the tool reconstruct descriptors without advancing through the entire workload again. Separate streams distinguish partition selection from clustering selection and column-value generation.

Reversible generation makes result comparison possible

A compact descriptor is useful only if the checker can relate it to returned data. Harry uses a bijection that inflates a descriptor into a typed value and deflates that value back into its descriptor. This lets validation translate database results into the same compact representation used by the model.

The simplest case in the article is a long-valued identity mapping. Expressed as Java methods, the operation is:

long inflate(long descriptor) {
    return descriptor;
}

long deflate(long value) {
    return value;
}

Strings and composite keys require more work. The article describes building strings from ordered pieces selected by portions of a descriptor. Signed long comparison and byte-wise string comparison differ, so preserving order requires handling the sign bit. Composite keys divide the descriptor's bits among component generators and combine the deflated components to reconstruct the descriptor.

Order preservation matters for more than equality checks. Clustering order and range deletions depend on how keys compare. If the generator's descriptor order disagreed with the generated key order, a model using descriptors could calculate the wrong range even while reconstructing individual values correctly. Harry's ordering requirement supports the model's interpretation of those operations.

Validation compares two independently obtained states

For a selected partition, Harry queries Cassandra and deflates the returned keys and values. It obtains a compact representation of the observed partition state. Separately, it replays the generated operations into a model using Cassandra's reconciliation rules. The model retains row descriptors and timestamps needed to determine the expected values.

Comparing those states checks whether the result matches the generated history. The checker does not need to reproduce each large original value in the model because its descriptor identifies that value. This is how compact generation becomes compact verification, rather than merely a smaller workload log.

The quiescent checker still needs to know whether each operation executed. An unresolved operation outcome prevents reconstruction of one definite expected history. The article explicitly excludes unknown events from this model. That limitation matters when designing failure-injection scenarios, where lost responses can leave execution uncertain even after a client stops waiting.

I regard that stated boundary as part of the tool's value. Harry can generate varied histories and verify their results, but its checker has an explicit condition on the histories it can interpret. Sequential or concurrent runners can supply data for quiescent validation when the state of every operation is known. The same model cannot silently treat an unknown outcome as an operation that never happened.