Transient replication separates Cassandra quorum size from full copies

Increasing a replica count usually means keeping another full copy of the data. Cassandra's transient replication work separates those decisions by allowing some replicas to retain only temporary data. The design can enlarge the group participating in availability decisions without requiring every member to store the repaired dataset indefinitely. Incremental repair supplies the mechanism that makes that separation possible.

The community's December 3 introduction presents transient replication as an experimental feature being developed for 4.0. CASSANDRA-14404 describes its objective as decoupling storage requirements from consensus-group size. The proposal is relevant to operators who want additional availability without an equal increase in full data copies. Its experimental status and compatibility limits are part of that objective's current scope.

I find Cassandra attractive when active-active operation at scale is a real requirement. Transient replication interests me because it makes the cost of replication more explicit. It also changes the obligations attached to repair, so its capacity argument needs to be understood together with its correctness mechanism.

Full and transient describe a range

A transient replica is temporary storage for a particular replicated range. It can receive writes when sufficient full replicas cannot provide the required acknowledgements. Once incremental repair propagates that data to full replicas, the transient copy can discard it.

These roles do not divide the cluster into full-storage nodes and transient-storage nodes. The introduction states that every node fully replicates some ranges and transiently replicates others. A node's role depends on the range being considered, which matters when reasoning about the workload after another node becomes unavailable.

Replica selection extends Cassandra's existing walk around the token ring. The last configured number of replicas found for a range are designated transient. With three total replicas and one transient replica, the selected replica set remains three members, but only two retain full copies of the repaired data.

Configuration expresses both quantities in a replication-factor value such as 3/1. Its first number is the total replica count, including transient replicas. Its second is the transient count. Misreading it as three full copies plus one temporary copy would produce the wrong redundancy calculation.

The following statement uses the syntax shown in the introduction for an existing keyspace and a datacenter named DC1. It describes the developing implementation, with its associated repair and compatibility requirements.

ALTER KEYSPACE foo WITH REPLICATION = {
  'class': 'NetworkTopologyStrategy',
  'DC1': '3/1'
};

The article also describes support for SimpleStrategy. Existing tables must have monotonic reads disabled through read_repair = 'NONE' for the implementation being introduced. After altering the keyspace, the documented sequence includes incremental repair followed by nodetool cleanup to remove data that should no longer remain on transient replicas.

Cheap quorums avoid unnecessary writes

The normal write path prefers full replicas. When enough full replicas can acknowledge the request, writing to transient replicas can largely be avoided. The design calls this optimization cheap quorums because transient replicas do less write processing and have less subsequent compaction work.

Avoiding an initial write to a replica creates a latency problem when one of the selected full replicas is slow. The coordinator may wait for acknowledgements that do not arrive promptly, even though another replica could accept the mutation. Rapid write protection responds by sending mutations to additional replicas after a delay.

The introduction describes the table option additional_write_policy as controlling that delay. The default uses the observed ninety-ninth-percentile latency. This is a mechanism for deciding when to involve more replicas, not a claim that all writes finish within that latency.

The design also sends writes to enough transient replicas to reach a quorum in every datacenter, although it need not wait for all such deliveries unless the requested consistency level requires them. That distinction separates sending an operation from waiting for its acknowledgement. It is necessary to understand what the client response establishes.

Transient storage is therefore not guaranteed to remain empty during normal operation. The article estimates that rapid write protection sends about one percent of writes to transient replicas under the default policy. Prolonged node unavailability creates additional transient write load. Those figures describe the design's expectations and default behavior, without establishing a storage bound for a particular deployment.

Repair makes temporary storage temporary

Incremental repair is responsible for propagating transient data to full replicas. During that repair, transient replicas stream out their data but do not receive data. Anti-compaction separates transient and fully replicated data so that the transient portion can be removed after repair completes.

This is the essential connection between availability and storage savings. A transient replica can help accept writes during a failure, but the design ultimately relies on moving those writes into the full copies. Discarding the temporary copy is justified by that completed propagation.

The introduction states that after incremental repair, full replicas for a range contain a synchronized repaired dataset. That permits any full replica to supply the repaired portion of a read. Unrepaired data still needs the read process described by the replication scheme.

Operationally, the article requires regular rolling incremental repair. A long interval allows more temporary data to accumulate, especially while a full replica is unavailable. Frequent repair keeps each transfer smaller and lets the temporary storage recover its intended role.

I regard that requirement as part of the capacity model. A projected reduction in disk use depends on the system completing the work that removes transient data. Treating repair as unrelated background housekeeping would omit the mechanism responsible for the savings.

The issue's description relates the design to witness replicas used in other distributed systems. Cassandra's version uses incremental repair to reconcile replicas that do not hold the complete dataset. The comparison is useful for understanding the objective, but the actual cleanup and read rules remain those of this implementation.

Reads still need a full replica

A transient replica cannot answer for repaired data it has already discarded. Reads must therefore include at least one full replica, together with enough additional replicas to satisfy the requested consistency level. This rule explains why a low consistency level cannot simply choose any replica without regard to its role.

The introduction explicitly identifies ONE as an exception to its general claim about the number of unavailable replicas tolerated before queries fail. That exception follows from the requirement to include a full replica. A reachable transient replica does not necessarily possess the older repaired data required for the answer.

The current implementation also does not support monotonic reads for transiently replicated keyspaces. The article describes a possible tension between cheap quorums and a future monotonic-read implementation. That future support is a development direction, without a guarantee about its eventual form.

Other compatibility boundaries are substantial. The announcement lists batch log, lightweight transactions and counters as unsupported work intended for later development. It identifies secondary indexes and materialized views as features it does not intend to support with transient replication. A cluster can contain both ordinary and transiently replicated keyspaces, so these restrictions can be applied per use case.

Failure count and failure magnitude differ

The most important qualification in the introduction concerns data loss. It says the number of permanent replica losses required before data loss is possible remains unchanged. It then distinguishes that threshold from how much data a particular set of losses can remove.

With 3/1, two replicas hold the repaired dataset in full. Permanently losing both can lose the entire repaired dataset for the affected ranges, because the transient replica has discarded those repaired copies. Counting three replicas without distinguishing their retained data would conceal that consequence.

A deployment using two datacenters with 3/1 in each retains four full copies in total. The article uses that arrangement to illustrate a different redundancy calculation. It does not establish that all topologies or failure patterns have equivalent consequences.

The operational advice also connects transient replication to vnode counts. A small number of vnodes spreads takeover work across several nodes without the problems the article associates with large vnode counts. Future cluster growth needs to be considered alongside that selection, since the range-role arrangement affects where transient work appears.

The intended audience is expert Cassandra operators able to investigate failures and contribute fixes. The experimental designation permits incremental development across releases, with compatibility gaps stated in advance. For that audience, the design provides a concrete storage option whose behavior depends on range-specific roles, full-replica participation in reads, and regular incremental repair that transfers and then removes temporary data.