rqlite can cancel cluster startup when a signal arrives

A database process that cannot form its cluster still needs to stop when asked. Issue 1746 reported that rqlite failed this expectation during startup. The reporter ran the process in a foreground Docker container with bootstrap and join options. While cluster formation was failing, neither an interrupt nor a normal container stop produced the expected exit. The report described resorting to a forced kill.

rqlite 8.23.4, published on April 27, addresses signal handling during that startup work. The release changelog identifies cancelable boot and join operations as the relevant fix. This is a small patch with a concrete operational consequence. A node's attempt to become useful can no longer assume that its original startup goal remains valid indefinitely.

I value rqlite's modest deployment requirements. That includes ordinary process behaviour as much as its single-binary packaging. An operator should be able to abandon a failed cluster-formation attempt through the same shutdown mechanism used for a healthy process. Needing a successful startup before shutdown becomes effective makes diagnosis harder at precisely the wrong time.

Connecting signals to the work already running

The change in pull request 1762 creates a main context from the signal channel near the start of the program. A goroutine waits for a signal and cancels that context. Cluster creation receives the context, which it then passes into explicit joining and bootstrapping. The signal now has a path to work performed before the main program reaches its normal steady-state wait.

Previously, installing signal handling at startup did not by itself make all intervening operations interruptible. A signal channel can receive a value while the main control flow remains inside a retry loop. Unless that loop observes a cancellation condition, the request to stop has no effect on its decisions. The patch carries that condition into the code that owns the retries.

The program's later wait also changes to observe the main context. Automatic backups use that same context. These changes give the process a shared cancellation condition across several activities. They do not require each activity to read directly from the signal channel, which would make signal consumption compete across independent consumers.

The source also shows why cancellation should not be described as an instantaneous interruption of every instruction. The join loop checks the context before attempting a target. Its existing join call still performs the individual communication operation. The patch changes the retry control flow without demonstrating that every underlying network operation accepts the context directly.

Retrying only while startup is wanted

The joiner now receives a context as an argument. Before each target attempt, it checks whether cancellation has occurred. If it has, the method returns a distinct join-canceled error. This gives its caller a reason for termination that differs from simply exhausting the configured attempts.

Between attempts, the old implementation slept for the retry interval. The replacement waits on either cancellation or the interval expiring. This is a material difference when the process receives a signal during a pause. It can leave the retry wait without waiting for the entire interval, then report cancellation through its normal return path.

Bootstrapping receives similar treatment. Its main select includes the context's completion channel alongside the existing timer events. When cancellation arrives, it checks the boot completion predicate before deciding the outcome. If the required condition has already become true, bootstrapping reports completion. Otherwise, it records a canceled status and returns the corresponding error.

That completion check handles a timing detail that a simple cancellation branch would miss. A cancellation notification and a successful cluster condition can coincide. The code does not report an incomplete boot merely because cancellation was the select case it observed. It consults the condition that defines success before choosing its final status.

Tests distinguish cancellation from failure

The pull request includes a bootstrap test with an already canceled context and an incomplete boot condition. It checks both the returned error and the canceled status. Another test supplies a completed condition while cancellation is present and expects success. Those cases exercise different interpretations of the same notification.

A join test cancels a context while the joiner has repeated attempts available. It expects the cancellation error rather than the generic failure returned after exhausting attempts. This verifies that cancellation can end retrying before the original plan runs to completion. The tests are useful because a process eventually exiting would be a much weaker assertion about this behaviour.

The release also contains increased use of atomic integers, as the changelog records. That is separate from the startup cancellation fix. The operational story here is the path connecting a shutdown signal to a join or bootstrap loop that would otherwise continue pursuing cluster formation.

I regard that path as part of the database's operating interface. Cluster startup can fail for reasons an operator needs time to investigate, and stopping the current attempt is often the first useful action. In this patch, bootstrapping and join retries can observe that decision through the context passed down from the process's signal handler.