February 2, 2021·4 min read·engineeringcockroachdbsecurityavailability

CockroachDB's HTTP decoder bug can bypass cluster redundancy

Advisory 58932 describes an unauthenticated HTTP request that can crash a node. Repeating the software failure across nodes can make the whole cluster unavailable.

Replication can tolerate a failed machine while leaving a cluster vulnerable to the same input crashing every machine. CockroachDB's advisory 58932 describes that second problem. A specially constructed HTTP request can trigger a bug in protobuf binary decoding and crash a node, even without an authenticated DB Console session.

The February 2 advisory classifies the issue as a security vulnerability because an attacker can cause full-cluster unavailability. It identifies CVE-2021-3121 as the underlying protobuf vulnerability. Kirk Baird receives credit for reporting the issue to Cockroach Labs.

I accept consensus overhead in exchange for a database that preserves consistency through node failures. That trade does not imply that every failure is independent. An HTTP input processed by the same vulnerable software on several nodes can defeat the availability benefit of maintaining several replicas.

Decoding happens before the useful authentication boundary

The advisory's decisive qualification is that no authenticated console session is needed. Requiring a console login therefore does not establish that an untrusted request cannot reach the faulty decoder. The request-processing path itself contains the failure.

The fix changes the decoding routine for protobuf payloads. That description identifies the implementation layer involved without suggesting that ordinary SQL permissions control the attack. A role's authority to create a database or modify a table is not the relevant prerequisite in the published report.

The disclosure does not provide a public request payload or identify an endpoint in its description. There is no reason to fill those gaps with a guessed exploit. The established mechanism is sufficient to understand the response, since reachability of the HTTP interface permits the unauthenticated input to arrive.

This distinction also matters when separating transport security from request safety. The advisory recommends limiting who can reach the port. It does not claim that adding a password to the console causes the decoder to reject the crafted input.

The HTTP listener is a separate interface

The 20.2 startup reference as available on February 2 defines --http-addr as the address and port for DB Console HTTP requests. Its default port is 8080, using the address portion of --listen-addr.

The same reference separately documents SQL listening and advertising addresses. A network policy written around the SQL service therefore does not, by its existence alone, describe exposure of the HTTP service. An assessment has to identify which listener the advisory concerns.

Port 8080 is the documented default, not a guarantee about a particular deployment. A configured --http-addr can change it. The security question concerns the actual HTTP endpoint, including any network path that forwards requests to it.

When upgrading is not possible, Cockroach Labs recommends auditing the network configuration to ensure the HTTP port is unavailable to untrusted clients. It specifically recommends placing the port behind a firewall. That action reduces the set of clients able to submit the crashing input.

A console login page remains an application-level control on an interface that is reachable. A firewall can instead prevent an untrusted source from reaching that interface at all. The advisory's unauthenticated trigger explains why those controls have different effects.

Replica count does not isolate a shared software failure

The replication architecture documented before this disclosure organizes each range's replicas into a Raft group. Committing a write requires agreement from a majority. Three replicas can tolerate one unavailable replica.

That guarantee depends on the remaining replicas continuing to operate. A vulnerability that crashes a node through a reachable interface can be directed at another node with the same code. Replication does not remove that shared implementation.

The advisory's full-cluster impact is therefore consistent with a database designed to survive individual node loss. The attacker creates correlated failures through a common input path. Adding another replica with the same exposed decoder does not repair the decoder.

I find that distinction useful when reading availability claims. The failure model matters as much as the number of machines. A design that handles a host disappearing still needs security maintenance for inputs that every surviving host can receive.

The immediate effect described here is loss of availability. The advisory does not claim data theft or data corruption from this request. Those are different outcomes and should not be attached to a crash vulnerability without supporting evidence. The published impact here is sufficient to justify the availability response.

Fixed versions define the update target

The mitigation section names 19.2.12, 20.1.11 and 20.2.4, or later releases, as upgrade targets. Its affected-version statement includes releases through 19.2.11, 20.1.10 and 20.2.3. These paired lists identify the maintenance boundaries relevant to the production branches in the disclosure.

For a deployment on 20.2.3, the named fixed maintenance target is 20.2.4. The other pairs serve the same purpose for their respective branches. This keeps the update decision tied to the version actually running. A cluster whose nodes have different installed versions also needs that distinction at node level, because the advisory describes a request that crashes an individual process. The surviving processes do not repair a vulnerable peer's decoder.

The 20.2.4 release notes already available by the advisory date date that release to January 21. February 2 is the advisory's publication date. The disclosure and the binary release are therefore separate events.

That timing gives an operator two useful records. The installed version determines whether the advisory's fix boundary has been crossed. Network exposure determines whether untrusted clients can reach a still-vulnerable HTTP listener.

A firewall mitigation changes reachability while the software update changes decoding behavior. Both follow directly from the published mechanism. The intended result is that an unauthenticated HTTP request cannot make the database process crash through this decoding bug, regardless of whether the rest of the cluster could survive one isolated node failure.