Cassandra partition denylisting trades selected access for cluster health

A pathological partition can consume resources that other requests need. Cassandra 4.1's partition denylist gives operators a way to reject access to selected partitions while leaving other data available. It deliberately reduces the accessible dataset to protect the service that handles the remaining requests.

Jordan West's June 30 introduction describes the feature for the upcoming release. Large partitions, excessive rows, and tombstones can make particular requests expensive enough to affect unrelated work. The denylist changes the response to known problematic keys into a failure. It does not remove those keys or correct the data model that produced them.

I regard this as a useful operational control for Cassandra's distributed architecture. Keeping every partition accessible at any cost is not always the right availability policy. The decision should be explicit, with a clear account of which operations the database will refuse.

Operators identify the partition to block

The administrative interface uses the StorageProxy JMX MBean. Adding an entry requires a keyspace, table, and partition key. The announcement demonstrates these commands in jmxterm, using foo.bar and the key baz as an example.

bean org.apache.cassandra.db:type=StorageProxy
run denylistKey foo bar baz
run isKeyDenylisted foo bar baz

The second operation checks the current denylist status. A separate removeDenylistKey operation removes an entry when the operator wants access restored. These are administrative actions. The client issuing an ordinary data request does not decide whether its target belongs on the denylist.

The database can deny reads, writes, or range reads that include a blocked partition. Those controls are separate, so the policy need not be identical for every operation. For example, an operator choosing to block writes is making a different intervention from one choosing to reject reads of existing data. The feature provides those choices without claiming that one policy fits every problematic partition.

The announcement's client example shows an invalid-query error for a blocked key while a different key remains readable. It also demonstrates a rejected insert when write denial is enabled. That response identifies the partition and table involved. An application can distinguish this explicit administrative refusal from a request that merely waits until its timeout expires.

Persistent entries feed local caches

The implementation in the 4.1 alpha tag stores entries in system_distributed.partition_denylist. It groups the denylisted keys for each table into a CQL partition. Each node maintains an in-memory view for enforcement, so evaluating an ordinary request does not require a fresh database query for the entire denylist.

Each cached entry holds immutable collections of keys and tokens. Refresh replaces the entry as a whole instead of mutating its collections. The keys support partition-specific checks, while the token collection represents the information needed when reasoning about ranges. Persistence and enforcement are consequently distinct parts of the feature. A stored entry must reach the relevant node's active cache before that node can use it.

The announcement supplies an explicit refresh operation.

bean org.apache.cassandra.db:type=StorageProxy
run loadPartitionDenylist

This operation acts on a node's cache. The configuration comments describe the expected usage as changing the state and refreshing caches, with periodic asynchronous refresh as a fallback. That is an important distinction for an intervention intended to stop expensive requests immediately. Changing the underlying entries is only part of getting an updated policy into effect.

The article describes a five-minute default refresh interval, but the alpha's configuration file specifies denylist_refresh: 600s, and its implementation comments describe ten minutes. For the alpha implementation discussed here, the tagged configuration is the concrete reference. Explicit refresh also avoids depending on an assumed periodic interval when applying an urgent change.

The master switch is separate from the operation switches. The alpha configuration documents partition_denylist_enabled as false, while read, write, and range-read denial settings are true. Those operation defaults describe behavior once the feature is enabled. They do not mean that a default installation is already enforcing partition blocks.

Loading policy has an availability tradeoff

The configuration exposes denylist_consistency_level, with QUORUM as its documented default. A stronger read requirement can make the loaded policy more dependable, but it also requires sufficient replicas to answer. West's explanation explicitly contrasts that with lower consistency, which improves the chance of loading while increasing the possibility of missed or stale entries.

This dependency matters because the feature exists partly for degraded conditions. The implementation checks whether enough nodes are available before attempting its initial load. If the load cannot proceed, it schedules another attempt. The configured initial-load retry interval in the alpha file is five seconds. Loading the safety policy thus has its own failure handling rather than assuming that every startup can immediately read the distributed table.

The implementation comments distinguish a manual JMX load from the initial and timer-driven checks. A manual call warns about unavailable consistency but proceeds with an attempt. That behavior gives the operator a way to try loading during degraded cluster conditions. It does not guarantee success or establish that every node now has identical cache contents.

The configuration also bounds the number of loaded keys. The alpha file documents limits of 1,000 per table and 10,000 in total. These limits constrain the administrative mechanism itself. A denylist intended for selected problematic partitions should not silently become an unbounded copy of application keyspace membership in memory.

Removal restores policy rather than repairing data

Removing an entry reverses the administrative block, but it does not by itself make the partition smaller or change the queries that access it. The original reason for blocking therefore remains a separate operational issue. Denylisting provides time and control over access while that issue receives attention.

I prefer the feature's explicit failure to allowing a known expensive partition to keep consuming shared resources without an administrative limit. The cached design also makes the operational responsibility clear. The policy exists in distributed storage, while each node enforces its own loaded view. Adding or removing an entry must account for those caches, and the read consistency setting determines what the loader requires from the cluster while obtaining that view.