CockroachDB 2.1 adds changefeeds and a managed operating model

A transactional database often feeds systems that cannot perform their work inside its transaction. A search index needs changed documents, a cache needs invalidation, and reporting may need a separate representation. Having an application write to both destinations creates another failure boundary. CockroachDB 2.1 introduces changefeeds as a database-managed way to expose row changes for downstream processing.

The October 30 release notes classify change data capture as an Enterprise beta feature, with row-level feeds into Apache Kafka. The release also coincides with Managed CockroachDB, initially offered with limited availability. These are different additions. One changes how other systems receive data, while the other changes who operates the database cluster.

I appreciate CockroachDB's serializable transaction model, but downstream systems need their own interpretation of changes. A transaction guarantee inside the database does not automatically establish the state of a separate search index or report. A changefeed makes that integration explicit, giving it a source and a progress model that can be reasoned about separately.

A feed starts with state and continues with changes

The 2.1 CREATE CHANGEFEED documentation describes a consistent scan of watched rows followed by subsequent changes when no cursor is supplied. A cursor instead starts after a specified timestamp without first emitting current table state. Those modes serve different purposes, establishing an initial downstream representation or continuing from a known point.

For an existing table named events, the documented Kafka form can be expressed as:

CREATE CHANGEFEED FOR TABLE events
INTO 'kafka://broker:9092'
WITH updated, resolved;

This requires the Enterprise capability and an authorized caller, as well as the Kafka destination. The command returns a job identifier. The updated option includes update timestamps, while resolved requests progress timestamps. These pieces give the consumer more information than a stream of values alone.

An initial scan matters when the downstream system begins empty. Receiving only future modifications would omit rows that already exist and never change again. Conversely, starting an entirely new scan when continuing an established feed changes the work the consumer receives. The cursor option makes that distinction an explicit part of the request.

The application therefore needs to know whether it is initializing or continuing its representation. The two operations may use the same table and sink but have different semantics. A consumer that treats every new feed as continuation can miss initial state. A consumer that treats every continuation as first-time initialization can repeat work unnecessarily.

The documentation supplies job operations for pausing, resuming and canceling a feed. It also describes finding a previous job's high-water timestamp to start another feed from that point. Progress is therefore an operational object as well as a property of individual records. The consumer and operator need to preserve the point from which continued processing should begin.

Downstream effects still need application semantics

Exposing row changes does not define what a downstream service should do with them. An indexer needs a rule for converting a changed row into its search document. A cache needs a key mapping and an invalidation policy. Those are application decisions even when the database supplies the feed reliably under its documented behavior.

The changefeed also does not turn the downstream service into a participant in the original SQL transaction. The source transaction and the external effect remain separate operations. A user reading directly from CockroachDB and a user reading a derived index can therefore ask different questions about the same underlying record. The integration needs a stated expectation about when the derived representation is useful.

My preference is to make that separation visible in the data model and the interface. The transactional database remains the source of the row. A derived representation serves a particular query or processing purpose. Connecting them through an explicit feed makes the maintenance of that representation inspectable without pretending the two systems share one commit operation.

The beta classification matters here. It identifies the maturity the project assigns to the capability in this release. It should remain in an account of the feature, since a reader planning an integration needs to distinguish availability from the project's assurance about readiness. Enterprise licensing is another independent condition that affects whether a deployment can use it.

Managed operation changes responsibility

Cockroach Labs' October 30 launch statement announces Managed CockroachDB on AWS and Google Cloud Platform. The release notes describe qualified customer acceptance under limited availability. They list provisioning, backups, upgrades and monitoring among the responsibilities provided by the service.

This addresses a different adoption cost from SQL compatibility. A compatible driver helps the application communicate with the database. It does not provision nodes or maintain an operational process for them. A managed service moves some of that work to the vendor, while application requirements still determine the desired topology and workload behavior.

The service does not remove the physical distances involved in a multi-region cluster. A transaction's communication still depends on placement and coordination. The operational benefit is that another party manages the cluster under the service arrangement. It is useful to describe that benefit directly instead of interpreting managed operation as a change to transaction physics.

For the application owner, the remaining decisions include which data belongs near which users and what failure behavior is required. Those requirements need to be communicated whether the cluster is self-managed or vendor-managed. Outsourcing operations changes who carries out procedures. It does not supply the application's meaning for locality or acceptable recovery.

Migration and planning gain their own interfaces

Version 2.1 also extends IMPORT to accept PostgreSQL and MySQL dump files. This removes a particular conversion step when bringing data into CockroachDB. It does not establish that every source application has identical behavior on the target. Data import and SQL compatibility remain related but separate migration questions.

The release enables the cost-based optimizer by default and adds experimental statistics generation and inspection. A cost-based choice depends on estimates about the data and available operations. Exposing statistics makes some of that evidence accessible to the person investigating a plan. The resulting plan can be evaluated against the query instead of being treated as an unexplained consequence of the new version.

EXPLAIN ANALYZE executes a query and returns a physical plan with execution statistics. That gives observed execution a distinct place alongside estimated planning. The distinction matters because an estimate expresses the optimizer's model, while runtime observations describe a particular execution. A discrepancy can direct attention toward the model or toward changing data and workload conditions.

The release notes also describe pipelining transactional writes during replication and disk writing. That change targets the execution of transactions performing multiple writes. It is a separate mechanism from choosing a better relational plan. A plan can determine which rows an operation visits, while the write path determines how the resulting modifications are coordinated and persisted. Keeping those explanations separate matters when interpreting the release's performance claims. A workload that spends most of its time in an inefficient read plan has a different opportunity for improvement from a transaction whose cost is dominated by several successive writes. The new optimizer and write pipelining address different parts of those operations.

I find those additions consistent with the practical appeal of SQL compatibility. Keeping familiar query interfaces is useful only if users can investigate the plans those interfaces produce. Better import paths make migration more accessible, while inspection tools help determine how the migrated workload actually runs. They address different stages of adoption without making performance claims for an unspecified application.

New capabilities retain explicit boundaries

The release removes SNAPSHOT isolation as a distinct option and maps requests for it to SERIALIZABLE. That is a behavior change a migrating application needs to understand. A familiar setting name can remain accepted while its resulting semantics change. Version-specific documentation matters when a client sets transaction options during initialization.

The changefeed and managed-service additions deserve the same precision. Changefeeds are Enterprise beta row subscriptions into Kafka. Managed CockroachDB is a limited-availability operating arrangement on the announced cloud platforms. Keeping those descriptions intact lets an application owner decide which new capability addresses a specific requirement.

For a downstream processor, the immediate design work is to define initial state, continuation and the meaning of the external effect. For a managed deployment, it is to specify the application's topology and operational expectations. CockroachDB 2.1 supplies new mechanisms for carrying out those decisions, while the application remains responsible for what the resulting data and service behavior mean.