December 16, 2019·4 min read·engineeringcockroachdbstoragereliability

CockroachDB 19.1.6 keeps load-based splits outside SQL rows

CockroachDB 19.1.6 fixes load-based splits that could divide a row's column families. It also restores a storage check before liveness heartbeats and carries qualified changefeed repairs.

A keyspace split can be valid as a division of bytes while violating a SQL row's representation. CockroachDB 19.1.6 fixes load-based splitting that could divide the column families of one row across ranges. The documented symptoms appeared in SQL as encoding-related panics, including reports of null values in non-null columns.

Tobias Schottdorf's safe-split repair adds a call to EnsureSafeSplitKey during load-based splitting. The changes delivered in 19.1.6 include this fix alongside storage-liveness and changefeed corrections.

I value CockroachDB's ability to distribute data while keeping a relational interface. That depends on boundaries between the SQL representation and the key-value layer remaining correct. This release repairs a case where a storage decision needed more information about the row encoded within the keys.

A row has a boundary below SQL

The split repair explicitly identifies splitting between column families of the same row as illegal. Load-based balancing therefore cannot choose an arbitrary key merely because it appears to divide a busy portion of the keyspace.

The change makes the split decision consult the safe-key mechanism. That preserves the representation expected by the SQL reader. The user-visible error could look like a problem in a column value even though the underlying cause was an invalid range boundary.

This distinction is useful when interpreting the release. A panic saying that a non-null column contains null does not necessarily mean an application inserted an illegal null. The commit identifies a lower-level representation failure that can manifest through such an error.

A follow-up split-decider adjustment handles errors from the safe-key check. The commit explains that these can mean the candidate is not a SQL key, in which case declining every split would be wrong.

Together, the changes preserve two requirements. SQL rows need safe split boundaries, while non-SQL keys must remain eligible for ordinary splitting. The storage layer cannot assume that every candidate has exactly the same higher-level interpretation.

A live process may have an unusable disk

Another repair writes to local storage before updating liveness. Irfan Sharif's commit explains that a stalled disk could otherwise coexist with successful liveness heartbeats.

Other nodes could then continue treating the node as live and avoid taking over its leases, even though it was unresponsive for the relevant work. Process activity was insufficient evidence of the storage service's ability to proceed.

The commit restores a previously introduced protection that had been lost through a sequence of later changes. Its purpose is specific. Local storage progress should be established before the node renews the liveness state that influences other nodes' takeover decisions.

Secondary logging receives an independent memory-leak repair in this release. Buffers allocated through secondary loggers accumulated on free lists that subsequent allocations did not use. The commit identifies SQL audit logging, statement execution logging and RocksDB logging as affected paths, making the fix relevant to installations using those diagnostic streams.

This concerns the meaning of a heartbeat. An availability mechanism needs an observation tied closely enough to useful work. A process that can update a remote liveness record while its local disk is stalled can provide misleading evidence about its ability to serve data.

My preference is for health mechanisms to test the dependency they represent. This fix puts a storage operation before the liveness update, directly connecting the advertised state with progress on the node's local storage path.

Changefeed fixes retain a compatibility limit

Version 19.1.6 also contains several changefeed repairs involving schema changes and restarts. One fixes a poller path that could emit a resolved timestamp before emitting row updates at that timestamp. Another repairs table-history initialization after a restart so the poller can detect required backfill work.

These are data-delivery correctness issues. A resolved timestamp is useful to consumers only if the feed respects the progress it represents. Restarting a job also has to retain enough schema history to recognize which rows still require emission.

The release does not carry every cloud-storage sink correction without qualification. A backport adjustment explicitly reverts file-naming changes deemed unsuitable for a patch release.

That commit says some invariant violations remain because their fixes require changing the output filename format. It recommends that users of the experimental cloud-storage changefeed sink upgrade to 19.2 or later. This is a present recommendation recorded before today's release, not a claim about an undisclosed future fix.

The maintenance release therefore has both implemented repairs and a stated compatibility boundary. Its safe-split changes protect SQL row representation, the liveness check connects advertised health with local storage progress, and the changefeed backport documents where preserving a patch release's output format limits the corrections it can include.