rqlite stops race tests at the first detector report

A race report near the beginning of a long test log can be followed by enough output to obscure the evidence. The build may eventually fail correctly while the maintainer still has to search for the conflicting accesses that caused it. Stopping the process at the first report changes that diagnostic experience.

Philip O'Toole's 26 September 2018 post explains the halt_on_error race-detector option and links to rqlite's CI configuration. He says he now enables it in his Go unit testing. The relevant change is immediate termination after a report, not the invention of a nonzero result for a race.

I like rqlite's reliance on Go and SQLite, but its background coordination still requires careful concurrency reasoning. A useful test report should identify the access pair and the goroutines involved without making the maintainer reconstruct that evidence from a much longer failing run.

Instrumentation and termination are separate controls

The Go 1.11 race-detector documentation describes -race for testing and for building programs. The flag creates an instrumented execution in which the detector can observe conflicting memory accesses. Setting GORACE without building or testing with instrumentation does not supply that instrumentation.

The environment variable configures the detector's behavior. Its documented halt_on_error default is zero, and enabling it causes the program to exit after reporting the first data race. This is separate from the exitcode option, which controls the status used when exiting after a detected race.

An illustrative command for a Bash invocation is short.

GORACE="halt_on_error=1" go test -race ./...

The package pattern chooses what to test. The flag enables observation, while the environment setting changes what the process does after its first finding. Keeping those roles separate helps explain why ordinary test success and race-test success provide different evidence.

The rqlite configuration linked by O'Toole applies the option as an environment setting beside a race-enabled test command. Its race job has its own timeout and remains distinct from the ordinary package-test job.

A reported race already fails Go tests

The testing implementation in Go 1.11 checks the race count when assessing test failure. It also checks for detected races when determining the final result of the test run. A run with a race does not become successful merely because the detector continues executing after its report.

This matters when describing the purpose of the CI option. An existing race-enabled job already has failure semantics. Enabling immediate termination reduces the work and output that follow the first report inside the affected process.

The scope is also narrower than canceling an entire CI workflow. A detector exits the instrumented program in which it finds the race. Separate jobs may still be running, and the setting does not itself specify a policy for canceling them.

For a package test process, immediate termination may prevent later tests from running. That is the intended tradeoff. The first report receives attention sooner, while other findings may require another execution after the first defect is repaired.

The report contains a relationship among accesses

The versioned detector documentation shows a report with the stack for a conflicting read and the stack for a preceding write. It also includes the stacks where the relevant goroutines were created. Those details connect the accesses to the concurrent execution that made them possible.

A normal assertion can pass while those accesses race. The observed final value is only one result of the execution. The detector supplies evidence about synchronization that a final-value assertion does not express.

That distinction is especially useful in tests manipulating shared state. A test may prepare a fixture while background work is already reading it. If the fixture modification bypasses the component's locking rules, the test itself participates in the conflict.

Stopping at the report helps preserve the immediate context of that failure. It does not determine the correct repair. The conflicting operations still need to be understood in terms of ownership and synchronization, including whether the test arranged a state that the production interface would never allow.

Runtime observation has a coverage limit

The documentation says the detector only finds races on paths that execute. A clean instrumented run therefore establishes a property of the exercised behavior. It does not prove that every possible execution of the program is free of data races.

Building a program with instrumentation can extend observation beyond unit tests, and the documentation describes running such a binary under a realistic workload. That remains a different execution with its own coverage. The unit-test option does not imply that rqlite's separate system-test job automatically uses an instrumented daemon.

The configuration makes the distinction inspectable. The race job invokes Go tests with -race, while the end-to-end job builds and launches the daemon separately. Each provides evidence about the code and execution it actually runs.

For me, the useful practice in O'Toole's post is keeping the first concurrency failure prominent. The detector already supplies a failing result, and halt_on_error=1 shortens the path from that finding to inspection of its stacks. The remaining test coverage can run again after the access relationship in that first report has been corrected.