CockroachDB 19.2 overlaps the consensus work of committing

A distributed transaction can wait for consensus twice even when its individual writes replicate concurrently. CockroachDB's earlier commit path first waited for the writes, then replicated the decision that the transaction had committed. CockroachDB 19.2 changes that dependency so the transaction record and outstanding writes can complete their consensus work in parallel.

The November 12 release announcement, available in the company's preserved release post, introduces Parallel Commits. Nathan VanBenschoten's November 7 protocol explanation supplies the mechanism behind the reduction in sequential consensus rounds.

I accept consensus latency when it protects transaction correctness. The interesting part of Parallel Commits is that it changes the dependency between durable operations while retaining the requirement that the writes and the commit information survive failure. The protocol reduces waiting without making acknowledgment depend on unreplicated changes.

The earlier path serialized two kinds of durable state

CockroachDB represents provisional writes as intents associated with a transaction. Its transaction record provides a way for another operation to determine whether those writes should be visible.

In the earlier path described by the protocol article, the coordinator laid down intents while processing mutation statements. Transactional pipelining allowed it to overlap write replication with continued interaction with the SQL client.

At commit time, however, the coordinator had to wait for those intent writes to become durable. Only then could it update the transaction record to the committed state. Replicating that state required another consensus round after the writes had completed.

The transaction record was replicated for the same availability reason as the data. A coordinator failure should not leave all other operations unable to discover the transaction's outcome. Removing replication of the record would trade away a property the system needs.

The latency problem therefore came from ordering the two durable steps. Writes to different ranges could already achieve consensus concurrently. The later transaction-record update still created a sequential dependency on their completion.

This matters even when the SQL statement looks small. The article uses secondary indexes to explain how an insert can become a cross-range transaction. Adding the first secondary index could move the insert away from a single-range fast path and expose the extra commit round.

STAGING changes the commit condition

Parallel Commits introduces a transaction-record state named STAGING. That state records the outstanding writes the transaction intends to complete. It is not, on its own, proof that the transaction committed.

The new commit condition combines the staged record with evidence that all recorded writes successfully replicated. The coordinator can therefore replicate the staged record while the outstanding writes are still reaching consensus. It no longer needs to wait for them before starting that record update.

Once the coordinator knows that the record and all the required writes succeeded, it can acknowledge commit to the client. The consensus operations overlap, reducing the sequential consensus portion of the path to one round in the described case.

This is a distributed condition. Under the earlier definition, a committed transaction was identified by one committed-state record. Under the new definition, the staged record and the recorded writes together can establish the outcome.

The distinction is necessary for correctness. Treating every staged transaction as committed would expose writes that might never have succeeded. Treating every staged transaction as aborted would discard transactions whose required state had already become durable.

The protocol also retains interactive transactions. A client need not declare the entire write set before beginning. As statements execute, the coordinator learns the writes and records the outstanding set when commit arrives.

That requirement matters for ordinary SQL applications, where a read can determine a later write. A protocol that demanded a fully declared write set at the start would impose a different application interface. Parallel Commits works within the existing interactive model described by the article.

Another observer needs a recovery procedure

The coordinator has direct knowledge of its outstanding writes, but another transaction encountering an intent does not. If it finds the corresponding transaction record in STAGING, it must establish whether the distributed commit condition holds.

The transaction status recovery procedure checks the recorded intents. If all required writes replicated successfully, the observer can conclude that the transaction committed. If a required intent is missing, it must also prevent that write from succeeding later before concluding that the transaction aborted.

The article identifies the timestamp cache as the mechanism used to prevent that missing write from later succeeding. That step matters because an abort conclusion must remain valid after the observer has made it. Merely observing a momentary absence would not be enough.

After determining the result, the observer can rewrite the record with a consolidated committed or aborted status. Future observers can then use the simpler record state instead of repeating the distributed investigation.

This recovery path is more expensive than the common coordinator path. The implementation tries to avoid it by having coordinators asynchronously move records to committed state and by using transaction heartbeats to identify a coordinator that is still active.

Those heartbeats let a conflicting transaction wait for a live coordinator to finish its work. Recovery is primarily intended for cases where the coordinator dies, rather than as routine overhead added to every successful transaction.

My preference is to assess this optimization through that failure path as well as its successful timeline. Overlapping writes is straightforward to describe. The substantive protocol work is preserving a determinate outcome when the process that knew those writes' status disappears.

The performance claim has a defined component

The release post describes reducing consensus rounds from two to one. That should not be read as a promise that every SQL transaction has half its former end-to-end latency.

Read statements still have their own latency. Contention can create other work, and execution can involve costs beyond consensus. The protocol article specifically identifies uncontended cross-range transactions as a class expected to benefit.

Its secondary-index experiment illustrates the removed dependency. The earlier latency increase when an insert first became cross-range disappears under Parallel Commits in that experiment. The explanation ties the result to a change in protocol path, rather than claiming that secondary indexes have no cost.

This is useful when interpreting the release's performance language. The reduced sequential consensus work is a mechanism that can improve single-region and multi-region deployments. Its share of total transaction time determines how much of that improvement appears in an application's response time.

Vectorization addresses work inside query execution

Version 19.2 also enables vectorized execution by default for eligible queries. The October 31 engineering article explains a different source of overhead, involving row-at-a-time interfaces and repeated interpretation of value types.

The vectorized approach processes batches and uses concrete typed operations. Working on columns within a batch improves memory access and amortizes the cost of moving between operators. Code generation produces type-specific implementations from templates so fixes need not be applied manually to every operator.

These changes do not replace distributed commit. They address CPU work within execution, especially for operations that process many rows. A transaction can benefit from a shorter commit path while an analytical query benefits from a different execution engine.

The engineering article distinguishes its toy multiplication benchmark from end-to-end query results. It reports a much larger improvement in isolated in-memory processing than in complete queries, where reading row-oriented data from storage remains part of the cost.

That distinction also prevents a mistaken storage conclusion. Column-oriented processing within the execution engine does not mean the database has replaced its stored row representation with a column store. The article explicitly describes reading data in its original row-oriented format before processing.

Placement and administration also become more explicit

The release adds control over backup locations for particular records, intended to support regional storage placement and reduce transfer costs. It also improves zone-configuration setup and visibility into table and index partitions.

These capabilities concern the operational expression of a distributed layout. They sit alongside protocol and execution improvements without sharing the same mechanism. Better visibility into partitions helps an administrator understand where data is arranged, while Parallel Commits changes how a transaction reaches a durable outcome.

I find the commit protocol the most consequential change because it targets a cost intrinsic to distributed transactions. It preserves replicated commit information and interactive SQL while overlapping consensus work that previously had to occur sequentially.

The result is a new normal path with a correspondingly explicit recovery procedure. A staged record establishes the intended write set, successful replicated writes complete the distributed condition, and observers retain a way to determine the outcome if the coordinator fails before consolidating the transaction record.