October 29, 2018·5 min read·engineeringcassandraauditingoperations

Cassandra 4.0 audit logging makes record loss a configuration decision

Cassandra's audit logging design puts user and request context into a configurable logging path. Its important tradeoff concerns what happens when the database produces records faster than the logger can persist them.

A database audit log has to account for the requests made against the database even when recording them becomes expensive. That requirement conflicts with the usual effort to keep work out of the request path. Cassandra's audit logging design addresses the conflict through explicit configuration and a pluggable logger. Its most consequential behavior concerns overload, when writing the record can delay the request that produced it.

Vinay Chella's October 29 description of audit logging for Cassandra 4.0 explains the implementation contributed by Netflix under CASSANDRA-12151. The work is part of the developing 4.0 version. It should be read as an account of that implementation, without implying that a production 4.0 release is available today.

I consider Cassandra's distributed availability model a strong reason to use it where the application needs it. Audit logging adds a different requirement to that decision. A system can remain available for requests while producing incomplete evidence about who made them, so the recording policy deserves its own design discussion.

Query capture and auditing ask different questions

The original audit issue requests the username, remote address and timestamp together with the action and targeted table. It also asks for connection attempts and changes to users or roles. That scope explains why ordinary query capture is insufficient. Requests that fail, and activity concerning access itself, can matter as much to an auditor as successful data mutations.

Cassandra already has full query logging work under CASSANDRA-13983. Its purpose is correctness testing through capturing production traffic for replay against different versions. The issue describes an asynchronous writer with bounded memory and disk use. That machinery gives the audit implementation a useful starting point, but the goals remain different.

Chella explains that full query logging instruments mutation and read paths while omitting context needed for auditing. The missing information includes failed queries, their origin and the user issuing them. The existing logging behavior also favors avoiding delay, which is reasonable for a testing aid but can conflict with an audit requirement.

This distinction matters when interpreting a log's completeness. A collection of successful CQL statements does not by itself establish a history of access attempts. Nor does a record useful for workload replay necessarily answer which identity attempted an operation. The audit design attaches those concerns to the request-handling path.

The queue cannot remove the overload tradeoff

The full query logging issue describes a single thread writing records asynchronously to disk. A weighted queue limits memory in front of that writer. If the queue fills, producers can block or samples can be dropped. Disk use is bounded by deleting old segments once a configured size is reached.

Asynchronous writing changes where the work happens, but the queue remains finite. If requests generate records faster than the writer can persist them for long enough, the producer must eventually encounter that limit. Buffering can absorb a temporary mismatch. It cannot make a sustained mismatch disappear.

The audit article makes this choice explicit. Dropping a record can preserve request performance, but the loss can violate the intended audit policy. Blocking preserves the opportunity to record the event at the cost of delaying the request path. Configuration has to express which behavior the installation requires.

That is an architectural constraint, not a defect specific to the queue library. A bounded logger cannot simultaneously promise unlimited intake, no delay and no loss under arbitrary load. The useful implementation question is whether the operator can see and choose the behavior at the capacity boundary.

I prefer that choice to be explicit because audit completeness is a property people may rely on outside the database team. A setting that allows record loss changes the meaning of the resulting evidence. It belongs in the same discussion as which requests are selected for recording.

Context and output are separate concerns

The design introduces AuditLogManager and IAuditLogger as abstractions joining audit logging with the existing logging work. The default implementation is BinAuditLogger, using the binary logging facilities built around OpenHFT's Chronicle Queue. A custom implementation can be placed on Cassandra's classpath and selected in configuration.

The supported event context includes the user when available, the execution host and source address. It also includes the source port and timestamp, request type and category, keyspace, scope and CQL operation. Those fields describe both the requested activity and the context in which Cassandra received it.

Scope is broader than a table name. The article identifies functions, aggregates and triggers among the possible targets. This is useful because auditing database activity includes schema and administrative operations as well as reading or changing application rows.

Pluggability permits different outputs without requiring the event-capture mechanism to be rewritten. It does not make all logger implementations equivalent. A destination and implementation still determine buffering, persistence and failure behavior. Selecting a custom logger therefore also selects operational behavior beyond the shape of the event fields.

Filters define the record being collected

The described YAML configuration can include or exclude keyspaces, categories and users. The article lists categories including QUERY, DML, DDL, DCL, AUTH, ERROR, PREPARE and OTHER. These filters let an installation narrow the activity it records.

That flexibility has a direct consequence for interpreting an audit file. Absence of an event may mean the configured policy excluded it. Completeness must be evaluated against the selected population of requests, as well as the logger's overload behavior. A record-loss setting and a filter answer different questions, even though both can remove events from the resulting log.

The article describes enabling auditing for a node with the following command, using YAML defaults. It also documents command-line filter overrides, so node-local changes can alter the selection used by that node.

nodetool enableauditlog

The command belongs to the developing 4.0 implementation discussed in the article. It is not a command added to a 3.11 maintenance release. Its node-level operation also means that enabling it once should not be interpreted as evidence about the configuration of every node.

Request context and an explicit recording policy give the design its practical value. The event fields can explain who attempted a CQL operation and where it originated. Configured filters determine which operations enter the log, while the selected logger and capacity settings determine how those records reach storage.