September 25, 2020·4 min read·engineeringkafkaconnectdatabases

Confluent Cloud adds managed database sources and warehouse sinks

The September connector release adds managed SQL Server and Oracle sources alongside Redshift and Snowflake sinks. Their direction and data-capture model determine what each integration contributes.

Moving data into Kafka and materializing Kafka records elsewhere are different integration jobs. Confluent Cloud's 25 September release notes make four managed database integrations generally available, with that distinction visible in their names. SQL Server and Oracle Database are JDBC sources. Redshift and Snowflake are sinks. The announcement broadens managed integration without making every database connection bidirectional.

The stated cloud coverage differs too. Redshift Sink launches for Kafka clusters on AWS. The SQL Server and Oracle sources and Snowflake Sink launch for clusters on AWS, Azure and Google Cloud. These are conditions of the announced managed offerings. A database name in a catalog does not by itself establish which direction or cloud combination is available.

I value Kafka's retained history because several systems can consume it without taking records away from one another. Database sources and warehouse sinks make that model more useful when they preserve a clear account of how records enter and leave the log. Managing the connector service reduces operational work, while the data movement still needs defined semantics.

Sources establish what enters the history

The Kafka 2.6 Connect guide distinguishes source connectors that import data from sink connectors that export it. A connector divides work into tasks, and tasks perform the copying. For a relational source, the mapping must include a way to identify progress in the source system. The existence of a Kafka offset does not automatically supply that source cursor.

The JDBC label in the announcement is consequently informative. Confluent's JDBC source configuration from the 5.5.0 release describes polling-based modes. Bulk mode reloads the table on each poll. Incrementing mode uses a strictly increasing column to discover new rows, and explicitly does not detect changes or deletions to existing rows.

Timestamp mode uses a column updated by writes to discover new and modified rows. The combined timestamp-and-incrementing mode adds a unique identifier to distinguish progress. These are the contemporaneous JDBC implementation's mechanisms, not a claim that every self-managed option is exposed identically in the managed service. They explain why a JDBC source should not be described as automatically reproducing every database change-log event.

That distinction matters for the history an application believes it has captured. Discovering new rows by an increasing identifier gives a different record stream from discovering modifications through timestamps. A connector can reliably execute its configured polling model without that model representing every kind of change the application cares about.

Sinks derive another system's state

Redshift and Snowflake appear as sink integrations in this release. That direction means Kafka supplies the input records and the external system receives their results. The announcement establishes managed availability, but it does not state that the two sinks have identical write behavior, schema rules or delivery guarantees. Those details cannot be inferred from the fact that both destinations support analytical workloads.

Kafka's 2.6 design documentation describes consumers choosing an offset in each fetch request and being able to rewind. That allows another consumer to derive a view from retained records. It also means replay is an application decision with consequences for the destination. Reading the input again and writing the destination again are related operations with separate behavior.

A sink therefore needs an account of what happens when records are processed again. Managed operation does not by itself establish that a destination write is harmless to repeat. The release note announces the service, while the integration's own write contract determines whether repeated inputs append rows, replace values or have another effect. The data model must agree with that behavior.

I find this separation useful because it keeps responsibility for meaning visible. The provider can manage the connector execution environment. The application still chooses whether the resulting external representation is appropriate for its records, including any replay it intends to perform.

Availability is narrower than a general database capability

The same dated release entry lists Azure Functions Sink and MySQL Sink as previews. They should not inherit general-availability status from the adjacent database announcements. Likewise, a source connector for SQL Server does not establish that the managed SQL Server sink has the same status. Direction and release status identify the actual offering.

This gives the September milestone a practical scope. Several commonly used database paths now have a generally available managed implementation. The useful architectural decision remains which path matches the application's movement of data. A JDBC source defines how database rows become Kafka records, while a warehouse sink defines a destination for records already in the log. Managed availability supplies an execution option for those paths without erasing their different progress and write semantics.