April 22, 2022·4 min read·engineeringrqliteconsistencytesting

rqlite's consistency tests depend on the read level

A Jepsen-style investigation reports linearizable behavior with strong reads enabled. rqlite's selectable read levels explain why that qualification belongs in the result.

A database test result needs to identify the consistency mode used by its clients. On April 22, Philip O'Toole points to Jepsen-style testing of rqlite by Nienke Eijsvogel, Ruben van Baarle, and Daan de Graaf. His summary reports linearizable insertion and modification behavior when strong read consistency is enabled. The qualification matters because rqlite exposes several read policies with different costs and guarantees.

I value the ability to select consistency per request. A report that names the selected mode is more useful than an unconditional claim that a distributed database is consistent. The application needs to know whether a read consults only local state or participates in the mechanism ordering changes across the cluster.

The investigation records concurrent histories

The linked investigation describes five Debian virtual machines running rqlite 7.3.1. Its register workload issues reads, writes, and conditional updates concurrently, records operation intervals, and checks the resulting history with Knossos. Fault injection includes partitions, paused processes, delayed messages, and packet loss.

Those choices test behavior beyond a successful query against an idle leader. During a partition, a process can remain alive and reachable by its client while lose contact with enough peers to act authoritatively. A consistency check needs to distinguish a valid result from a result produced by a node whose local view no longer represents the cluster's accepted ordering.

The result remains evidence about the tested workload and configuration. It does not replace the contract of every available read mode. The report's strong-read condition connects its observed histories to the part of rqlite's interface intended to provide the required ordering.

A local read can observe an older database

The read-consistency documentation available before the report distinguishes none, weak, and strong. With none, a node queries its own SQLite database whether it is leader or follower. A replica can be behind the leader or disconnected from it, so a successful local query need not represent the newest accepted changes.

This is an explicit service choice. It avoids the extra coordination needed by stronger reads and lets a read-only node serve its local contents. An application using that mode needs to accept the corresponding possibility of stale data. The database's use of consensus for changes does not make every local read automatically consult consensus.

The freshness parameter provides a related but narrower control. For none reads, the node checks the elapsed time since its last contact with the leader and rejects the request if it exceeds the supplied duration. The documented check concerns contact time. It should not be interpreted as an independently verified timestamp for every value returned by the query.

Checking local leadership leaves a transition window

With weak, a follower forwards the query to the leader and returns its response. The leader checks its local leadership state before querying SQLite. This avoids serving an arbitrary follower's local data while keeping the leadership check inexpensive.

The documentation identifies a window between that local check and reading the database. Another node can become leader and accept changes during the interval. The old node's earlier leadership observation does not establish that it remained authoritative throughout query processing.

This is a specific distinction between checking a local role and coordinating an operation with peers. The former reports what the process believes at the check. The latter can establish ordering through communication. A client needs to choose based on the observation its operation requires.

Strong reads participate in consensus

The documented strong mode forwards requests from followers to the leader too, but the leader sends the query through Raft. This requires communication with at least a quorum and increases response time. The query can then reflect changes ordered before it through the consensus system.

The interface makes the selection visible in the request. For example, a client querying a table named registers can request the strong path explicitly.

curl -G 'localhost:4001/db/query?level=strong' \
  --data-urlencode 'q=SELECT value FROM registers WHERE id=1'

The same endpoint can serve other read policies by changing level. The documentation identifies weak as the default, so omitting the parameter does not request the mode used in the reported strong-read result. That default belongs in any comparison between a test configuration and an application's actual requests.

The selected contract determines how to read the result

The investigation's finding is useful because it ties observed behavior to a concrete mode. A client asking for a local read and a client asking for a consensus-ordered read have requested different operations, even if their SQL text is identical. Evaluating both under one unconditional consistency claim would erase the interface choice.

I consider that explicit choice one of rqlite's practical strengths. A request can pay for coordination when its result needs the stronger ordering. Another request can accept local state when that is appropriate to its purpose. The reported tests provide evidence for the strong path, while the documented distinctions explain why applications must preserve that mode selection in the requests whose correctness depends on it.