Confluent Cloud adds generally available search and lake sinks

A retained event log can feed a search system and an archive without making either destination responsible for delivering events to the other. That arrangement matters when their requirements diverge. A search application needs a representation suitable for its queries, while an archive needs data that remains useful for later processing. Kafka provides the history between the producer and those separate delivery paths.

The November 20 Confluent Cloud release notes announce general availability for two managed sinks. Elasticsearch Service Sink is available for Kafka clusters on AWS, Azure, or Google Cloud. Azure Data Lake Storage Gen2 Sink is available for Kafka clusters on Azure. These are connector availability changes, with distinct deployment scopes. The announcement does not establish that every possible destination topology or configuration is supported.

I value Kafka's separation of retained history from consumer progress. It permits different consumers to interpret the same records without making one consumer's successful processing remove another's input. Managed sinks extend that arrangement to external systems while moving connector operation into the service.

Moving the worker does not define the data model

Kafka's 2.6 Connect guide describes a framework for moving data between Kafka and external systems. It separates worker configuration from connector-specific settings. Converters handle serialized keys and values, and transformations can make lightweight changes to individual records. These distinctions help explain the responsibilities that remain when a service operates the connector workers.

The application still needs to decide what a source record means at the destination. Consider a hypothetical order topic containing several changes for one order. A search view might need the latest state of that order, while an archive might need every change. Those are different requirements even if both sinks receive identical Kafka records. Selecting a connector does not resolve which representation the application needs.

Record identity is part of that decision. An order identifier describes a business entity, while a topic partition and offset describe an event's location. Two changes for one order can share the former and have different values for the latter. A destination design must say whether it preserves both changes or combines them into one current representation. This is a question about the intended output, independent of the deployment location of the worker.

The distinction also affects recovery. Repeating delivery to a destination may be acceptable when the write replaces the same logical result. Repeating an append can create another result. These examples explain why an application cannot infer destination behavior from the presence of a managed Kafka consumer alone. The relevant connector's documented write semantics must match the intended representation.

Serialization remains part of the interface

Connect's converters give the integration a defined place to interpret the bytes in a Kafka record. The Apache guide documents separate key and value converters, including connector-level overrides. Its framework also allows a transformation chain to modify records before delivery. Those capabilities explain the difference between having access to a topic and knowing how to interpret its contents.

For example, a value that contains a timestamp as text has an application-level meaning that its bytes cannot establish by themselves. Consumers need agreement about the timestamp's format and intended interpretation. A destination that expects a different representation needs an explicit conversion rule. Changing that rule can change the resulting data even if the source history remains identical.

I regard this as part of the event contract. A durable record is useful for replay only when its interpretation remains available. Moving delivery into a managed service does not remove the need to retain the schema or other information required to read older records.

That requirement becomes visible when rebuilding an external view. Replaying an old record through a changed transformation can produce a different output. Sometimes that is the purpose of the rebuild, such as correcting a previous interpretation. In other cases it creates an unintended difference between historical and newly delivered data. The reconstruction procedure needs to name the interpretation it applies.

Two delivery paths need separate progress

Search delivery and lake delivery can advance at different rates. An application that reads from the search system has evidence about that system's view, not about the archive's completeness. Conversely, data arriving in storage does not establish that the search view contains the corresponding changes. Each destination needs a definition of what successful delivery means for its own readers.

This makes the two GA announcements useful in different ways. A managed search sink can support an independently maintained query representation. A managed lake sink can support a separate storage path for later processing. Their shared source lets the application reason about which events each path should contain without requiring the destinations to exchange data directly.

The Azure scope deserves explicit treatment in architecture decisions. November's entry makes the Data Lake Storage Gen2 sink generally available for Kafka clusters on Azure. Elasticsearch Service Sink has the broader three-cloud scope in the same entry. A deployment drawing that places both behind an arbitrary Kafka cluster would erase a real difference in the announced product availability.

For a system using both destinations, completeness must remain a property of each delivery path. The producer's success in writing an event to Kafka does not establish that either external representation is ready for its readers. Keeping those separate conditions visible allows the search application and archive processing to advance according to their own requirements.