rqlite 8.37.0 reloads TLS certificates when connections need them

Certificate renewal should not require restarting a database node merely to replace the identity it presents on a connection. rqlite 8.37.0, released on 1 May, adds automatic certificate reloads for the HTTP service and inter-node communication. The released mechanism checks the configured files when TLS asks for a certificate. It replaces a polling design developed earlier in the same release.

I value rqlite's relatively small operational requirements. Removing a restart from routine certificate renewal is consistent with that preference. The useful engineering detail is the scope of the change. It refreshes a certificate and private-key pair through callbacks without rebuilding the whole TLS configuration.

The implementation changed during development

The release changelog records work on a polling-based certificate monitor as well as later work on lazy loading. Reading the entries as independent descriptions of the final behavior would be misleading. They record successive implementation steps that do not all remain active in the released code.

PR #2080 removes the monitor and introduces CertReloader. The deleted monitor used a ticker and a background goroutine. Its callers had to start it and arrange to stop it when the owning component closed. The replacement removes those lifecycle calls.

That matters beyond code size. Polling ties observation to a timer whether or not a certificate is currently needed. The replacement ties observation to certificate retrieval. The implementation therefore does not promise that an idle process notices a changed file after a fixed interval.

The distinction changes how to describe deployment behavior. Replacing a file and waiting is not itself evidence that a new certificate has been served. The relevant path is a TLS operation that requests the certificate. The code then checks whether the files are newer than the pair it has already loaded.

Construction establishes a valid starting pair

The released reloader stores the certificate and key paths, a modification timestamp, and a pointer to the loaded certificate. Construction calls tls.LoadX509KeyPair through a small wrapper. A failure prevents successful construction.

After loading the initial pair, construction reads both file timestamps and stores the latest one. The cached pair and recorded time therefore begin together. This is a stronger starting condition than postponing all validation until the first incoming connection.

The runtime behavior deliberately differs from startup. Invalid initial files prevent the component from obtaining a usable reloader. A later invalid replacement can leave an already loaded pair available. This distinction allows renewal failures to preserve a previously working identity without pretending that an invalid initial configuration is acceptable.

It also makes the file paths part of the persistent configuration for the running process. The callback rereads those configured paths. The mechanism does not search for newly named files or obtain renewed certificates from an issuer. A separate provisioning process remains responsible for putting the intended pair at those paths.

Retrieval checks timestamps before parsing

Each call to GetCertificate acquires a read lock and obtains the latest modification time of the configured files. If neither file has a timestamp after the recorded value, it returns the cached certificate. The common unchanged case therefore avoids reparsing the pair.

When a newer timestamp appears, the method releases the read lock and obtains a write lock. It checks the timestamps again after acquiring that lock. Another caller may already have performed the reload while this caller waited. The second check prevents unnecessary duplicate work.

A successful reload replaces the cached pointer and advances the stored timestamp. The code updates those values while holding the write lock. Concurrent retrieval can therefore share the unchanged certificate, while replacement has an exclusive section.

The comparison is specifically about a later modification time. It is not a hash comparison or a file-content comparison. Replacing contents while preserving the recorded timestamps is outside the change-detection condition implemented here. The mechanism's simplicity relies on the provisioning process producing a detectable timestamp change.

This has a practical implication for file delivery. The files at the configured paths must become readable as a matching pair, and their observed modification time must advance. The database process can handle a transient bad pair, but it cannot infer a content change that its timestamp predicate does not recognize.

An unsuccessful replacement keeps the previous certificate

If reading file metadata fails, the method logs the failure and returns the prior certificate. If the timestamps indicate a change but loading the pair fails, it also logs the failure and returns the prior certificate. In either case, the failed attempt does not advance the stored timestamp.

That last detail permits a later retrieval to retry. Consider a provisioning process that replaces one member of the pair before the other. An intermediate load may see a mismatch. Retaining the old pair avoids replacing valid cached state with invalid material, and leaving the timestamp unchanged allows another attempt after the second file arrives.

The reloader tests include a mismatch case. They establish a working certificate, replace only the key, and check that retrieval still returns the earlier certificate identity. Other tests cover initial bad files, unchanged retrieval, and successful replacement.

Keeping the previous pair is a continuity policy with a specific limit. It does not renew that pair's validity or guarantee that peers will continue to accept it. A failed rotation can leave the process serving its old identity. The error log therefore describes a renewal failure even when connections continue working.

My preference is for this explicit retention behavior. Destroying the last usable pair because a replacement is temporarily incomplete would turn a provisioning race into an immediate connection failure. Returning the cached pair allows the provisioning process to finish while keeping the next retrieval eligible to retry.

Both sides of inter-node TLS need the callback

The TLS configuration helpers connect the reloader to Go's TLS callbacks. Server configuration uses GetCertificate. Client configuration uses GetClientCertificate, which matters when a server requests a client certificate for mutual TLS.

The development changes wire this mechanism into the HTTP server, the inter-node listener, and inter-node client creation. Updating only the listener would leave outbound client identity fixed. The client callback extends rotation to the certificate presented when the node acts as a TLS client.

The consequence is about certificate selection for TLS operations that invoke those callbacks. It is not a request to terminate every established connection. Nothing in the reloader walks active connections, replaces their state, or forces reconnects. Existing connections and new certificate retrieval are separate concerns.

That separation preserves the narrow purpose of the change. A certificate renewal can become visible without restarting the node, while the transport continues to manage its own connections. Forcing every connection to close would be a separate behavior with separate availability consequences.

CA trust remains outside this reload path

The configuration helpers still load a configured CA file into a certificate pool while building the TLS configuration. The reloader holds certificate and key paths, not a CA path. It therefore provides no basis for claiming that a changed CA bundle is automatically incorporated by the same mechanism.

That boundary matters when renewal and trust migration happen together. Issuing another leaf certificate under an already trusted issuer is a different operation from changing which issuers the process trusts. The released callback addresses the pair the node presents, while the CA pool follows the configuration-construction path.

The distinction also applies to verification settings and expected server names. Those remain inputs to the TLS configuration helpers. Reloading a certificate pair does not re-evaluate every security-related option supplied when the configuration was created.

The counters measure retrieval attempts

PR #2084 wraps server certificate retrieval with statistics. The HTTP service adds tls_cert_fetched, and the inter-node multiplexer adds num_tls_cert_fetched. Each wrapper increments its counter before calling the reloader.

Those counters therefore count callback retrievals, not successful file replacements. A retrieval with unchanged files increments the counter, and a reload failure that returns the previous pair also occurs after the increment. The reloader's successful-replacement log provides different information from the retrieval count.

For operators, this release removes process restart from the supported certificate-pair refresh path. The remaining contract is concrete. Provision matching files at the configured paths with advancing modification times, and certificate retrieval checks them before returning the current usable pair. Changes to CA trust still follow a different configuration path.