rqlite repairs Consul discovery over TLS

A database node can have working storage and still fail to find the cluster it should join. In rqlite's Consul discovery mode, finding that cluster requires a separate client connection to Consul. Securing that connection introduces configuration that must survive translation through a discovery library. The rqlite 7.13.2 release, published on February 23, repairs a bug in Consul-based autoclustering when Consul uses TLS.

The interesting part of this patch is where the correction enters the program. Pull request 1162 changes the version of rqlite-disco-clients in the Go module dependencies. It does not replace rqlite's discovery service or introduce a new clustering protocol. Following that dependency into its source explains the configuration involved and why an apparently peripheral client library can prevent a database from forming a cluster.

I like rqlite because its operational machinery is proportionate to the problem it solves. Automatic discovery can support that aim when node addresses are inconvenient to configure individually. It also adds an external dependency to cluster formation, and that dependency needs the same precise configuration treatment as the database's own listeners.

What Consul records

The discovery service in 7.13.2 has a small interface to its discovery backend. It can retrieve a recorded leader, initialize a leader record, and update that record. Its registration loop first asks whether the backend already knows a leader. If it does, registration returns that node's API address so the caller can proceed with joining.

If there is no recorded leader, the service attempts to initialize the record with its own details. The Consul client selected by this release implements that operation with Consul's compare-and-set API. The record lives beneath the configured discovery key, at a path ending in /leader. Its JSON value contains a node identifier together with its API and node addresses.

This is a discovery record. It supplies information that helps a node locate the cluster. The service separately asks the store whether the local node is leader before reporting an updated address. That ordering is visible in StartReporting(), where the store's leadership state controls calls to the backend's SetLeader() method. Consul does not become the component answering the store's leadership question.

The distinction helps explain the impact of the TLS bug. Failure to contact Consul interrupts the information exchange used by this discovery mode. It does not, by itself, demonstrate a fault in SQLite query execution. The affected connection needs to be identified before interpreting a startup failure as a database-engine problem.

Configuration crosses a library boundary

The node startup code creates a Consul configuration from the discovery configuration reader, then passes it to the discovery client constructor. Inside the dependency, consulConfigFromClientConfig() constructs HashiCorp's Consul API configuration. This is the point where rqlite's JSON fields become the settings used for actual requests.

The versioned configuration definitions separate the server address from its URI scheme. Its reader rejects addresses beginning with http, so a configuration should supply a host and port in address and put https in scheme. The following illustrates those fields together with a private certificate authority file.

{
  "address": "consul.example.net:8501",
  "scheme": "https",
  "tls_config": {
    "ca_file": "/etc/rqlite/consul-ca.pem"
  }
}

The hostname, port, and path are illustrative deployment values. The field names come from the dependency revision included in 7.13.2. Its translation function copies the address and scheme into the Consul API configuration. When tls_config is present, it also copies the CA settings and the optional client certificate and key settings.

Those certificate fields matter when Consul requires a client certificate. The configuration defines cert_file and key_file together, as well as PEM forms. It also exposes a verification override, but the CA configuration supplies a direct way to express trust in the server certificate without selecting that override. Which credentials a deployment needs depends on how its Consul endpoint is configured.

These are settings for the connection to Consul. Configuring them does not establish encryption policy for every connection in a database cluster. Keeping that scope explicit prevents a successful discovery request from becoming misleading evidence about unrelated traffic.

Retry behavior explains the startup symptoms

The registration loop logs errors from leader lookup and leader initialization, then waits before trying again. Its default registration interval is three seconds, with randomized additional delay. A persistent connection failure can therefore appear as repeated discovery messages while the node waits for registration to succeed. The loop explains why a bad external connection can dominate startup even though the configuration file itself parsed successfully.

The service also updates discovery information after leadership notifications and periodically. That supports recovery from intermittent failures that leave the recorded information stale. The code makes both responsibilities visible, with registration finding an initial destination and reporting keeping the destination current after the database's leadership state changes.

I regard this separation as useful operational information. It gives a precise question to ask of a failing node, namely whether it can complete the discovery request with the configuration its client actually receives. That question is more informative than assuming every cluster startup problem belongs to consensus.

The release changelog also lists a CLI connection-message improvement. Pull request 1156 recognizes connection-refused errors when the shell requests the server version and prints the configured protocol, host, and port. Other errors retain their existing message. This does not diagnose Consul, since it concerns the interactive client's connection to rqlited. It applies the same useful principle at a different connection boundary, by making the failed destination explicit enough for an operator to investigate.