CockroachDB slow-transaction tracing could put every node in a crash loop
A nonzero tracing threshold exposed an invalid operation on a no-op span. Advisory 68005 explains the affected releases and why an inherited setting can break an upgrade.
A setting intended to explain slow transactions can prevent the database from staying up long enough to investigate them. CockroachDB's advisory 68005 identifies that failure for sql.trace.txn.enable_threshold. Setting it to a nonzero value on affected 21.1 releases causes nodes to crash repeatedly.
The August 20 advisory names versions 21.1.0 through 21.1.6. The fix is available in 21.1.7 and later. It attributes the failure to misuse of an internal tracing API.
I want diagnostic controls to be usable when a distributed database is under pressure. That requires more than keeping their overhead manageable. Their state transitions must remain valid when tracing was previously doing no work, because that can be the normal starting state for an investigation.
A no-op span cannot become a recording span
The fix description and patch explain the API misuse. A transaction context could contain a no-op tracing span. The slow-transaction code then tried to make that span verbose, an operation the tracing API prohibited.
The fix forces creation of a real span whenever slow-transaction tracing is enabled. It moves the threshold check earlier in transaction setup so the decision can affect span creation. The later call to enable verbose recording then receives an appropriate span.
This is a state-precondition error. The code deciding to request more detail did not ensure that the object receiving the request supported recording. A setting that enabled tracing exposed the mismatch between the lightweight default object and the active recording operation.
The patch also preserves the case where session tracing is already recording. Both an existing recording request and a nonzero slow-transaction threshold cause the transaction to receive a real span. The choice is made before the code tries to enable verbose output.
The patch reads the configured duration before choosing how to create the transaction span. It tests whether recording is already active or the duration is greater than zero. Either condition selects the force-real-span option. This makes the prerequisite visible in the same branch that allocates the span, instead of relying on a later caller to discover whether recording is possible.
A zero threshold leaves the ordinary nonrecording path available when session tracing is also inactive. The fix therefore retains the distinction between disabled tracing and a request that needs a recording-capable object.
That ordering matters more than the size of the code change. Checking the setting after creating a no-op span leaves the incompatible object in place. Checking it first lets the creation path satisfy the later operation's requirement.
The setting makes the failure persistent
The advisory describes all nodes entering a permanent crash loop while the nonzero setting remains in effect. Repeated restarts do not remove the condition that sends transaction setup through the faulty tracing path.
A process supervisor can restart a crashed database process, but restarting the same binary with the same triggering configuration repeats the failure. The advisory's recovery instructions therefore change the software or recover a state without the triggering setting.
The disclosed impact is availability. It does not describe this tracing defect as an attacker reading data or corrupting table contents. The practical severity comes from making the cluster unusable through a diagnostic configuration.
Because the setting is cluster-wide, the problem is not isolated to a single observer's session. A diagnostic choice intended to collect transaction traces becomes an input used by the affected nodes. That explains why the disclosure discusses the entire cluster.
I consider this a useful distinction between session diagnostics and shared operational policy. A threshold applied throughout the cluster has a broader effect than a tracing request on one connection. The engine still needs to create valid recording state for every transaction that follows that policy.
An upgrade can inherit the trigger
The advisory explicitly covers upgrading a 20.2 cluster whose threshold is already nonzero. An operator does not have to change the setting after installing 21.1 to encounter the bug. The existing cluster configuration can activate it in the new code.
That makes configuration history part of upgrade compatibility. A setting accepted by the earlier branch can become hazardous when a new implementation reads it. An upgrade assessment limited to the new version's default values misses that path.
The advisory says rollback to 20.2 can mitigate such an upgrade only if the version upgrade has not been finalized. That is a concrete boundary on recovery, rather than a general promise that replacing the binary with an older one is always valid.
The 21.1 upgrade guide available at release time explains finalization separately from rolling out binaries. By default, finalization follows once all nodes run the new version. It enables capabilities that prevent downgrading to 20.2.
The guide documents cluster.preserve_downgrade_option as the means to delay that transition while assessing the upgraded cluster. This is a pre-upgrade control, not an instruction to reverse an already finalized upgrade.
Recovery follows the retained configuration
The primary remedy is upgrading to 21.1.7 or later. That changes transaction setup so the slow-tracing threshold creates a real span. It addresses the internal API violation even when the setting remains nonzero.
The advisory also describes backup-based recovery. One option is restoring a backup into a 20.2 cluster. Another is restoring into 21.1 from a backup without the triggering setting. The configuration inside the restored state therefore matters alongside the destination version.
Those options should not be reduced to restoring any backup on any version. A restored nonzero setting on another affected 21.1 binary preserves the failure condition. The advisory specifies the combinations that avoid it.
The patch contains a regression test that sets the threshold to one second and then back to zero. That test exercises the transition that previously crashed, rather than merely testing whether tracing functions exist. The repair ensures transaction setup creates a real recording-capable span before slow-transaction tracing tries to make it verbose.