CockroachDB fixes an incomplete node drain reporting success

A shutdown procedure depends on the meaning of its completion signal. If a drain command returns while work remains, the next step can terminate the process at exactly the point the drain was meant to avoid. CockroachDB 25.3.4, released on October 30, fixes that failure in cockroach node drain for nodes using virtual clusters, including deployments running Physical Cluster Replication.

The release describes a concrete consequence. The command could return before draining completed, allowing shutdown while SQL clients and range leases were still active. This was not simply an inaccurate progress display. A caller using command completion as permission to stop the process could act on the wrong state.

I regard maintenance behavior as part of CockroachDB's operational contract. Its replication model makes controlled transfer of responsibility possible, but the shutdown interface must wait for that transfer. A successful command is useful only when its meaning matches the state an operator or process manager needs.

A missing field hid an active draining stage

The implementation patch identifies the omitted state. While the server controller was draining, the response could fail to populate IsDraining. The CLI interpreted that response as indicating completion. Work in the server-controller stage was therefore outside the status calculation used to decide whether draining was still in progress.

Previously, the status helper checked whether SQL client connections were draining or the KV node was draining. The fix adds the server controller's own draining state. That extends the predicate to cover another component participating in the same shutdown operation, rather than changing the command into a fixed-duration sleep.

The patch's sample output illustrates successive kinds of remaining work. It first reports tenant servers, then reports liveness and lease-related work before reaching completion. Those counts describe different stages. A change in their magnitude does not by itself establish that shutdown regressed, because the next stage can introduce a different set of objects to finish.

There is a second adjustment in the patch. The decision to delay the first SQL client drain now checks the SQL server's draining state directly. It no longer relies on the broader predicate, which now also includes the server controller. That preserves the intended opportunity for readiness probes to observe the node becoming unready before client draining proceeds.

Draining transfers active responsibilities

The node shutdown documentation describes several ordered stages. A node first becomes unready so that a load balancer can stop directing new connections to it. Connection handling and SQL transaction draining follow, with a separate stage for distributed statements initiated through other gateway nodes.

These stages protect different requests. A node can serve as the gateway for one transaction and participate in distributed execution initiated elsewhere. Closing its client-facing connections does not alone establish that every other node has finished using it for execution. The distinction explains why shutdown needs more than a single count of local SQL sessions.

Lease transfer has another purpose. The draining node must transfer range leases and Raft leaderships to other nodes. Those responsibilities determine how work is served and coordinated, even though the range replicas themselves can remain on the machine during a temporary shutdown.

This differs from permanent removal. The node command reference distinguishes drain, which prepares a node for process termination, from decommission, which removes its replicas for removal from the cluster. A successful drain should not be interpreted as confirmation that decommissioning has occurred.

Completion remains distinct from a timeout

The CLI's --drain-wait sets the overall time allowed for manual draining. The command reference gives a default of ten minutes and says that a timed-out drain must be reinitiated to continue. Individual shutdown settings control particular stages, while the overall command deadline bounds the attempt as a whole.

Those limits cannot repair an incorrectly reported completion state. Increasing a timeout helps only when the caller continues waiting for genuine remaining work. The October fix restores that prerequisite by including the controller in the reported state, so its participation is visible to the waiting CLI.

I find the patch's regression coverage particularly relevant to its scope. The existing drain test now runs both with and without a secondary tenant, and the secondary-tenant case establishes a connection before draining. The test checks reported progress and remaining work in the configuration that exposed the missing state, rather than assuming single-server behavior represents every deployment.

Process supervision still has its own deadline. The shutdown documentation explains that an insufficient termination grace period can cause a process manager to send SIGKILL, bypassing graceful shutdown. Version 25.3.4 repairs premature completion reporting; the configured grace period must still allow the actual draining stages to finish before forced process termination.