CockroachDB 19.1.1 keeps deleted row keys in changefeed values
The key_in_value option makes a JSON changefeed value identify its row even after deletion. Cloud-storage sinks require that representation because they write values without a separate message key.
A deletion event needs to identify the row that disappeared. A null replacement value says that the row is gone, but it is insufficient if the consumer cannot recover the row's key elsewhere. CockroachDB 19.1.1 adds a JSON changefeed option that places the key inside the emitted value.
The 19.1.1 encoder implements key_in_value for the wrapped JSON envelope. Its cloud-storage sink requires the option. That requirement follows directly from how the sink writes records, since it emits the value without a separate message key.
I value a database's consistency guarantees most when they remain meaningful to the systems receiving its changes. A consumer cannot maintain a correct representation of a table if deletion messages lose the identity of the deleted row. This patch repairs a concrete part of that integration contract.
Message keys and row contents are different
The encoder interface has separate methods for keys and values. For JSON, the key is an array containing primary-key column values. The ordinary row contents are represented as a JSON object mapping column names to their values.
Those two representations can coexist in a transport that carries a key and a value as separate fields. A consumer can obtain the row identity from the message key even when the value describes a deletion. That arrangement is less useful when a sink persists only the value.
The cloud-storage sink's EmitRow signature makes the distinction visible. It accepts a key argument but does not use it. It writes the encoded value into a buffer and adds a record delimiter. Without embedding the key in that value, the persisted record would not retain the separate identity supplied to the sink.
This is a data-format problem at the destination boundary. The database still has the primary key, and the encoder can construct it. The missing step is preserving that identity in the bytes that the chosen sink actually writes.
The fix reuses the JSON key construction when encoding the value. It adds a top-level key entry alongside the wrapped envelope's after entry. The primary-key values therefore survive in the same persisted record as the row change.
This also makes the representation useful after the file leaves the original transport. A parser reading the stored JSON need not reconstruct a message key from an unavailable channel. The identity needed to interpret the deletion travels with its value.
Deletion preserves identity without promising old contents
The encoder's row structure documents a useful limit. For a deletion, only the primary-key columns are guaranteed to be set in the row data. A consumer should not infer that the deleted row's full previous contents are available merely because its identity is retained.
For an ordinary wrapped change, after contains the new row object. For a deletion, after is null. With key_in_value, both forms also carry the key array.
For a single-column primary key with value seven, a deletion has the following shape. This is an illustrative value derived from the tagged encoder, with object-member order immaterial.
{"after": null, "key": [7]}
The key is enough to identify which downstream row to remove. It is not a before-image of that row. Applications that require old non-key values have a different information requirement from applications that only need to apply the deletion.
The encoder constructs the key using the primary index's column identifiers, mapping those identifiers to the table columns. This matters for composite keys because identity is an ordered array of the actual primary-key values, not a single field guessed from a conventional column name.
My preference is for a changefeed format to make these distinctions explicit. Row identity, replacement contents and optional metadata have different purposes. Keeping them separate inside one envelope makes a deletion understandable without inventing data that the event does not carry.
The option has format restrictions
The JSON encoder rejects key_in_value unless the envelope is wrapped. This prevents accepting a configuration that cannot produce the documented representation. The Avro encoder also rejects the option in this release.
Cloud storage separately requires JSON and the wrapped envelope. Its constructor checks these conditions and requires key_in_value. The configuration checks align the sink's persistence behavior with the encoder's ability to retain identity.
The sink uses newline-delimited JSON files and groups buffered records by table name and schema version. It tracks timestamps for file handling and can write resolved-timestamp payloads separately. Those are implementation details of delivery, while the embedded key solves the row-identity issue within each change value.
This is why a small option is consequential for downstream consumers. A file of row updates is only useful for maintaining another dataset if deletions remain attributable. Version 19.1.1 gives the wrapped JSON value that information and makes the cloud-storage sink require it, without claiming that the same option is available in every output format.