July 24, 2023·4 min read·engineeringcassandraobservabilitycqlperformance

Cassandra 4.1.3 exposes running tasks through CQL

Cassandra adds system_views.queries with task descriptions and separate queue and running times. The implementation defines a local view of current work rather than a durable query history.

A latency metric says how long an operation took, but it does not necessarily identify the work still executing while a problem is happening. Cassandra 4.1.3, released on July 24, adds a virtual table exposing currently running query work. The release changelog identifies the addition as CASSANDRA-15241.

The table is system_views.queries. Its rows include a thread identifier, a task description, and separate queue and running durations in microseconds. Those fields make an important distinction visible. Work can spend time waiting for execution before it spends time running.

I find Cassandra's distributed model useful when an application needs its availability properties. That also makes local evidence valuable during diagnosis. Knowing what one node is doing can narrow a question that a cluster-wide latency distribution leaves unanswered.

The released schema is more precise than the proposal

The original issue asks for current queries and their duration. Its example uses a single duration_micros column. The 4.1.3 implementation instead defines queued_micros and running_micros.

That difference is why a query should follow the released table schema, not simply copy the proposal's output. The implementation separates two stages of a task's lifetime. A long queue interval and a long running interval represent different observations, even if both contribute to an application's wait.

The basic inspection statement is direct.

SELECT thread_id, queued_micros, running_micros, task
FROM system_views.queries;

The query reads runtime information through CQL. It does not start a repair, change a configuration, or cancel a task. The addition is a diagnostic view with a specific source of data.

The thread identifier is the partition key in the virtual schema. This identifies a worker in the reported view. It is not documented here as a durable application-request identifier that can be used to join every stage of a distributed operation.

The rows come from executor workers

The implementation walks running debuggable tasks in the shared executor pool. For each entry with a task, it reads creation and start times and obtains a task description. It constructs the result in memory when the table's data method runs.

The source comment describes coordinator work on the native transport request stage and local read or mutation work. This explains why the task column can contain different kinds of descriptions. A client query and the local work it causes are not necessarily presented as one identical row.

The example in the released source includes native transport, mutation, and read-stage entries. It even includes the query inspecting system_views.queries. Observing current work can therefore include the observer's own request.

This is a local execution view. The code traverses the process's shared workers, not a distributed catalogue of every node's tasks. A client connecting through an arbitrary coordinator should not infer that one result is a simultaneous inventory of all work across the cluster.

Timing describes the reported task

Queue time comes from the interval between task creation and its start, or the current time when no start is available. Running time comes from the interval after the start, with zero used when a start is unavailable. The implementation converts nanosecond differences to microseconds for the table.

Those calculations explain the fields without turning them into an end-to-end application latency measurement. A task's creation is the beginning of the interval known to this implementation. An application may have spent time before that task existed or may perform more work after it ends.

The two durations are useful for different investigations. A large queued value points to a task that waited before execution. A large running value identifies time accumulated after starting. Neither number alone proves which shared resource or query property caused the delay.

The description provides context for that investigation. It can identify a query or mutation-related task, but the source does not promise that every description has one universal text format. Tooling consuming the view needs to treat the released columns as the schema and avoid assuming every task string is a complete SQL statement.

Current work is not a historical record

A task that has already completed need not appear when the table is queried. Constructing the view from current executor state does not retain a history of finished operations. Short tasks may begin and end between observations.

My preference is to use this kind of table alongside historical metrics. The current view can identify work during an incident, while a latency series describes how behavior changed over time. Each supplies information the other cannot reconstruct by itself.

Cassandra 4.1.3 consequently adds a focused diagnostic capability. It exposes the shared executor's current debuggable tasks through a familiar query interface and distinguishes waiting from execution. The local scope and transient lifetime of those rows remain part of interpreting what the result says about the database.