Cassandra 4.0 beta makes workload replay part of release validation

A database release can pass its tests and still encounter a workload its developers did not represent. Cassandra 4.0 enters beta with an invitation to run application workloads in test and QA environments. Full query logging gives that invitation a practical basis. A recorded workload can be replayed against different database versions, with their results available for comparison.

The Apache release record dates the first 4.0 beta to July 20, 2020. The community announcement commits to no new features or breaking API changes in subsequent beta or general-availability builds. That makes the beta a useful point for application compatibility work. The announcement still directs readers to test and QA environments, so its confidence in the testing effort should not be rewritten as a production deployment recommendation.

I find Cassandra attractive when an application needs active-active operation at scale. That makes the ability to examine a real workload across versions more persuasive to me than a general statement about stability. The useful question is which behaviour a test represents, and what conclusions the comparison can support.

Capturing the request after it completes

The beta full-query-logging documentation describes a binary log backed by Chronicle Queue. Its records include query text and the options associated with execution. Batch records also include the values bound to the queries. Timestamp state and approximate invocation time provide information needed to describe how the recorded request ran.

The point at which logging occurs is a material limitation. The full query logger logs requests after successful completion. Failed requests and timed-out requests do not enter this log. A replay built from these records therefore represents successfully completed traffic. It does not, on its own, reproduce the complete set of requests the application attempted.

That distinction matters for interpreting differences between releases. A capture can provide evidence about the results of successful requests without representing how an application behaves after an error. The logging contract establishes that boundary before any comparison begins. Error paths need evidence that actually contains the relevant errors and subsequent actions.

The tools operate on different stages of the process. fqltool can dump a readable representation of the log or follow an active log. Its replay and compare operations support the broader testing workflow. Recording is therefore separate from examining the recorded traffic, issuing requests against another cluster and comparing results. Each stage can be inspected without treating the capture file as a complete test specification.

Logging consumes bounded resources

The implementation puts a weighted queue in front of a single asynchronous writer thread. Queue weight bounds the memory assigned to pending records. A separate limit governs the size of rolled log files retained on disk. These controls express how much production traffic the logger can retain without allowing storage demand to grow indefinitely.

When the queue fills, configuration determines whether producers block or records are dropped. The documented default for block is true. The choice changes both the completeness of the capture and the possible effect on query execution. Dropping records permits gaps in the recorded workload, while blocking makes producers wait for logging capacity.

This is a useful example of why a statement about asynchronous logging needs more detail. A background writer does not imply that application threads can never wait for it. The queue policy determines what happens when the background work falls behind. The beta documentation exposes that policy directly.

Disk retention creates a second boundary. max_log_size controls the rolled files retained before the oldest file is deleted. The limit describes retained rolled files, so it should not be casually presented as an exact bound on every byte associated with a running logger. Log rolling and reclamation are distinct actions in the documented implementation.

The roll cycle can be minutely, hourly or daily, with hourly as the default. An archive command can process rolled files, and the configuration includes a retry limit for that command. Those options allow retention outside the immediate queue, but they do not establish the properties of an external archive. Its destination and availability remain separate parts of a deployment design.

A dedicated log directory is also part of the contract. The documentation warns that full query logging can recursively delete the contents of its configured directory. A directory used for unrelated files would therefore give the logger authority over data outside its intended purpose. Naming a logging path is a decision about ownership as well as a location for output.

Comparison has assumptions of its own

The project described its validation methods in the earlier testing account. Replay testing compares a recorded workload on stable and candidate clusters. For a useful subset of workloads, grouping writes by CQL partition and ordering them by client-supplied timestamps gives deterministic execution. The article explicitly excludes complete coverage of every workload, including compare-and-set queries, from that technique.

The same account describes property-based testing against a model of legal database states and combining tests with injected failures. Upgrade testing begins with matching cluster copies, upgrades one and compares the resulting data. These methods answer different questions. A replay comparison does not replace a model of valid behaviour under failures, and a clean-version test does not establish a rolling-upgrade result.

That is the release argument I find useful. Confidence grows from evidence whose scope can be described. A test result is easier to assess when its input and comparison rule are explicit, and when unsupported workloads are named in the testing method itself.

Audit logging records a different kind of evidence

The beta audit-logging documentation describes login attempts and CQL commands attempted or successfully executed through the native protocol. It includes unsuccessful authentication. That scope differs from the successful-query capture used by the full query logger.

Audit records identify the user when available and include the host, source address and source port. Request type and category help describe what operation was attempted. Keyspace and scope identify its database context where applicable. These fields answer questions about access and activity that a workload replay does not necessarily answer.

Prepared statements expose another distinction. The audit log records the query supplied at preparation, but does not include the actual values bound during execution. A record can establish that a prepared operation occurred without providing the parameter values required to reconstruct its complete invocation. That is a documented limitation, not an inference from the word audit.

The beta provides a binary logger and a file logger, with the community recommending the binary implementation for performance. Configuration can include or exclude keyspaces, categories and users. Operators can enable audit logging with nodetool as well as through the YAML configuration, and command options can override configured filters.

Filters determine the evidence that remains available. The defaults exclude several system keyspaces, and the documentation warns that overriding the exclusions requires including those defaults again if they should remain excluded. An override is therefore a replacement of the setting in this case. It should not be interpreted as an automatic addition to an invisible set of defaults.

The beta also defines the migration being tested

The tagged upgrade notes identify compatibility work that needs representative stored data. They describe incorrect frozen-UDT information in serialization headers written by Cassandra 3.0. On detecting an upgrade from that version, the mitigation rewrites the affected headers. The notes explicitly say that snapshots and backups are outside that migration.

That difference matters to the scope of an upgrade exercise. Data files used by the running cluster and files held for recovery do not necessarily undergo the same changes. A comparison of the upgraded live database cannot establish that every retained backup has received the same repair. The release notes provide a specific reason to examine recovery inputs separately.

The notes also describe removing COMPACT STORAGE support and require migration of those tables in the earlier database version. Such a schema cannot simply be treated as an ordinary table whose read results can be compared after starting the beta. The upgrade has prerequisites, and application validation begins with the representation the new version can load.

Cassandra 4.0 beta makes capture and comparison more available while documenting the limits of both. Full query logging retains successfully completed requests under explicit queue and disk policies. Audit logging records a broader class of attempted activity but omits prepared-statement bound values. Those contracts determine which questions each log can answer when a workload is brought to the beta.