April 30, 2019·6 min read·engineeringcockroachdbreplicationlocalitysql

CockroachDB 19.1 makes historical reads aware of replica distance

CockroachDB 19.1 can send sufficiently old reads to nearby replicas and account for index locality during optimization. These features reduce network work under specific data and transaction conditions.

A replicated copy of a row is useful for local reads only when the database can establish which version it is allowed to serve. CockroachDB 19.1 adds follower reads for historical queries, allowing eligible reads to use a nearby replica instead of traveling to the leaseholder. The latency benefit comes with an explicit timestamp condition.

The April 30 release announcement describes queries using AS OF SYSTEM TIME with a sufficiently old timestamp. These can be directed to nearby replicas. A new function, experimental_follower_read_timestamp, produces a timestamp intended to qualify for that path.

I accept consensus round trips when they provide the consistency an application needs. That does not mean every read needs the freshest possible state. Historical queries offer a useful place to reduce distance, provided the application makes its freshness requirement explicit.

Eligibility is more specific than replica availability

The 19.1.0 follower-read implementation checks more than the existence of a replica. A batch must contain read-only transactional requests. The transaction must exist and must not be writing.

Those conditions prevent a historical-read optimization from becoming a general routing rule for arbitrary operations. A transaction performing writes has different requirements from a read-only request for a past version. The implementation keeps those cases separate.

Timestamp age is another condition. The code derives a duration from closed-timestamp settings and a configured multiple, then accounts for clock uncertainty in the routing threshold. The helper used by the SQL function returns a negative offset from the current time.

The relevant effect is that the request targets a sufficiently old point. It does not ask each replica to return whatever value happens to be present when contacted. The timestamp remains part of the requested database operation.

The implementation also checks whether follower reads are enabled and performs an enterprise-license check. Although the announcement's appendix places follower reads under a Core heading, the tagged implementation contains that check. Deployment decisions need to account for the actual release's gating, rather than relying on the appendix heading alone.

This is a useful limitation to state without turning the feature into a general warning about distribution. The optimization has a defined domain. Read-only transactional work at an eligible historical timestamp can take a routing path that current writing transactions cannot.

A timestamp changes the application's question

A request for a historical snapshot answers a different business question from a request for current state. That difference can be acceptable for reference information or a view that deliberately tolerates some age. It is less suitable when the operation's meaning depends on observing a just-completed change.

The release does not claim to remove that distinction. Its function chooses an acceptable timestamp for follower-read eligibility, not an acceptable freshness policy for every application. The latter remains a requirement of the query being built.

This makes the SQL interface useful. The freshness choice is represented in the request instead of being hidden inside a routing convention. A reviewer can identify that the query asks about a past version and evaluate whether that matches the application's purpose.

The routing code also distinguishes eligible and ineligible transactions when choosing its replica policy. For an eligible historical transaction, it selects a closest-replica policy. Otherwise, it retains the existing bin-packing policy.

That is a concrete implementation of locality preference. The database does not assume the same destination is best for every operation. It considers whether the transaction can use the follower-read path before applying the alternative selection rule.

My preference is for a latency optimization with this kind of visible semantic condition. It provides a way to avoid unnecessary distance without pretending that old and current data are interchangeable. The value depends on matching the timestamp to a real application requirement.

Index locality is a separate optimization decision

CockroachDB 19.1 also adds locality awareness to its cost-based optimizer. The announcement describes preferring indexes in the same locality for appropriate queries. Its example is read-only access to reference tables in a geographically distributed cluster.

This concerns selecting an access path, while follower reads concern which replica can serve a timestamped request. Both can reduce network traffic, but they operate at different decisions in the query process. Treating them as one feature would obscure their prerequisites.

The appendix says index-locality preference requires per-index zones, an enterprise feature, even though the optimizer capability itself is not classified as enterprise. That dependency matters when planning a deployment that expects a particular local index to be available.

An optimizer cannot benefit from a local representation that has not been arranged. Likewise, arranging data locality is not sufficient if planning ignores the difference between local and remote access. The release connects placement information with the cost model so the choice can reflect the cluster's geography.

Automatic statistics collection supplies another part of the planning model. The announcement contrasts collected information with earlier assumptions of one thousand rows per table and ten percent distinct values per column. Those assumptions could guide planning, but they did not describe the actual distribution of a specific table.

The combination is relevant to distributed query work. Locality describes where access occurs, while cardinality and distinct-value estimates describe how much work an access path may produce. Better placement awareness and better table information can influence different terms of the same planning decision.

The optimizer gains more control over joins

Join reordering allows the optimizer to investigate functionally equivalent alternatives. The announcement gives a default setting of four and says setting it to zero retains the order written in the query. This makes the search scope configurable.

Join hints add another form of control when the optimizer selects an inefficient algorithm. The release does not describe hints as a replacement for optimization. They provide an override for cases where the cost-based choice misses a useful alternative.

Correlated subqueries also gain an execution option. Building on earlier decorrelation work, 19.1 adds an apply operator for cases that cannot be decorrelated. It executes a subplan for each input row.

That expands the set of queries the cost-based optimizer can handle, but it does not imply that correlated work has become free. Executing a subplan per row is a specific mechanism. Its cost still depends on the input and the work performed by that subplan.

The release introduces a query-plan cache as well. Reusing optimized plans reduces repeated optimization work for recurring queries. This addresses planning overhead, which is distinct from reducing the runtime network distance of a historical read.

Security integration has a defined mechanism

The security additions include encryption at rest for enterprise users. The announcement says this can encrypt stored data without changes to client applications. That is a property of stored database data, separate from choosing a query timestamp or replica.

Directory integration uses GSSAPI/Kerberos authentication for LDAP or Active Directory environments. Describing it simply as LDAP integration loses the authentication mechanism named in the appendix. The intended benefit is using organizational credentials according to configured access levels.

These capabilities address deployment requirements around the database. They do not change which historical reads qualify for follower routing. Keeping the mechanisms separate makes it easier to assess the release against an application's actual requirements.

The announcement also expands enterprise change data capture and adds a cloud-storage sink. Watched-table changes can feed other systems without requiring transport through Kafka for that sink. An experimental PostgreSQL-protocol-based changefeed implementation is described separately from the production enterprise capability.

This separation between mature and experimental paths matters within a major release. A feature list can place related capabilities together even when their support expectations differ. The release's own descriptions provide the distinctions needed to evaluate them.

Distributed SQL still needs an explicit access model

The practical effect of 19.1 is broader than one faster read path. It gives planning more information about data and location, expands supported query forms, and supplies a timestamped route to nearby replicas. Each improvement addresses a different source of work.

I find the follower-read contract particularly useful because it names the tradeoff in terms the application can express. A query asks for a past version and becomes eligible for a different replica choice. The tagged implementation then checks read-only transaction state, timestamp age and the feature's enablement before allowing that choice.

The same precision applies to index locality. Per-index zones establish the placement that the optimizer can prefer, while statistics describe the data the plan will process. Version 19.1 brings these mechanisms into query decisions without making one uniform promise about the latency of every transaction in a geographically distributed cluster.