ksqlDB makes materialized stream state available to pull queries

A continuously maintained customer profile is less useful to an application if the application cannot look up one customer. KSQL could already derive and update tables from Kafka streams, but its query model continuously emitted changes. The November 20 ksqlDB announcement adds point-in-time pull queries over those materialized tables. It also adds management of Kafka Connect connectors and gives the combined system a new name.

The new query capability matters because the state already exists. Jay Kreps's announcement describes KSQL and Kafka Streams maintaining disk-resident state with RocksDB and sequencing updates through a commit log. The release exposes that maintained result to interactive lookups. A consumer of the derived data no longer necessarily needs a separate serving database merely to retrieve a current value by key.

I value Kafka as durable history that several systems can read independently. Materialized state is a useful counterpart to that history because most applications also need answers about the present. ksqlDB brings the update process and one way to retrieve its result into the same system.

Push and pull name different query lifetimes

The announcement calls the existing continuous queries push queries. They remain active and produce an ongoing feed as source data changes. A pull query returns a result representing the current materialized state when the query executes. The distinction concerns the lifetime of the request and the result it expects.

The launch example looks up a ride's current location through ROWKEY. With the table and columns already defined, its point lookup has this form.

SELECT ride_id, current_latitude, current_longitude
FROM ride_locations
WHERE ROWKEY = '6fd0fcdb';

Appending EMIT CHANGES requests the continuous form instead. The syntax makes the query's lifetime explicit where a client chooses to subscribe to updates.

SELECT ride_id, current_latitude, current_longitude
FROM ride_locations
WHERE ROWKEY = '6fd0fcdb'
EMIT CHANGES;

These are queries against the announcement's example schema, not a complete definition of the underlying materialization. The first requests a point-in-time answer. The second supplies a continuing sequence of answers as the location changes. Both depend on the table's update process having produced the state they expose.

That last condition matters. A query being written in SQL does not establish when every upstream event has been incorporated. Query lifetime and update consistency are different aspects of the interface. The launch explicitly says materialization is asynchronous and does not provide read-after-write consistency.

Queryable state was already a stream-processing requirement

The Kafka Streams 2.3 interactive-query documentation describes application state split across running instances and their local stores. A local store query exposes only the state available on that instance. Accessing the whole application's state requires discovering instances and communicating with the instance that holds the relevant portion.

The guide provides local query APIs and discovery information, but requires an application-supplied RPC layer for remote communication. That is substantial work for a program that only needs to serve a materialized answer. The stream-processing library supplies pieces of the state model while the application still has to expose a network service around them.

ksqlDB's launch adds a database-like interaction with derived state to a server-based system. The contribution is therefore broader than shortening a local-store API call. It gives the application's caller a query interface without asking that caller to assemble the same serving arrangement directly around Kafka Streams.

The underlying state remains distributed. A network query interface does not make all data local to every process. The launch's discussion of failover and availability reflects that fact. The serving operation depends on the system's assignment and recovery of the state needed to answer it.

The Streams guide also treats interactive store access as read-only. State should be changed through the processing topology and its inputs, so direct out-of-band mutations cannot invalidate the topology's state model. That is useful context for the new pull path. It is access to a result derived from a stream, not an instruction to bypass the stream and edit the maintained result arbitrarily.

Connector management brings input acquisition into the interface

The second launch addition concerns data that is not yet in Kafka. Kafka Connect already has connectors for obtaining data from external systems and delivering data to other systems. ksqlDB adds direct control of those connectors through its own interface. The launch illustrates this with CREATE SOURCE CONNECTOR and a JDBC source connector.

This reduces the number of management interfaces involved in defining an application. An operator can describe ingestion, continuous transformation and retrieval through the ksqlDB workflow. The connector still implements the communication with the source system. Management integration does not imply that SQL alone defines the source database's capture semantics.

The distinction matters because the announcement's connector example contains omitted properties. It illustrates the management command and a PostgreSQL connection URL, but it is not a complete runnable connector configuration. A finished application still needs the connector's required settings and the intended relationship between source data and Kafka records.

The architectural simplification is nevertheless concrete. A separate export pipeline and serving store are unnecessary for some applications that only need supported lookups over already maintained state. A connector can bring data in, persistent queries can update a derived table, and pull queries can retrieve that table's result. The launch combines these functions around Kafka rather than requiring an independently chosen product for every stage.

Asynchronous views have a different consistency contract

The launch explicitly excludes use as an OLTP primary store and states that read-after-write consistency is unavailable. That is a consequence applications must account for when retrieving derived state. A successful source write and a subsequent pull query do not automatically establish that the materialization has processed that write.

A user interface can therefore receive an earlier derived value even when the source system has accepted a newer event. This is an implication of the documented asynchronous model, not evidence that either operation failed. The application needs to distinguish source acceptance from incorporation into the view it queries.

The model is well suited to continually computed results whose inputs already arrive as events. It is less suitable as a replacement for a transactionally updated source of record that requires immediate visibility of its own writes. The announcement makes this boundary explicit instead of claiming that the new database label removes every difference from a conventional application database.

My interest is in reducing the amount of separate infrastructure needed to serve a derived answer. That benefit does not require treating the derived view as the authoritative source for the original facts. The stream can retain history while the materialization supplies an asynchronously updated interpretation of it.

Serving availability and query scope remain bounded

Kreps identifies failover time as a limitation on serving availability in this initial release. The announcement says recovery currently takes several seconds while waiting for full group availability. It discusses improvements as future work. That means the query-serving responsibility comes with the stream processor's recovery behavior, rather than the availability characteristics of a separate database chosen for that role.

The launch also describes broader pull-query expressions, including range queries and complex groupings, as planned expansions. Those plans should not be read as a claim that arbitrary analytical SQL is already supported. The concrete example is a key lookup over a materialized table, alongside continuous queries that update or expose changing state.

Another stated implementation cost is that multiple persistent processing queries compile into separate Kafka Streams topologies. The announcement identifies resource-consumption optimization here as future work. Combining functions in one product therefore does not mean the execution cost of several queries has already been combined into one optimal topology.

The announcement also retains a role for external serving systems with richer query requirements. It gives a search-indexing pipeline as an example, with ksqlDB assembling documents and sending them to Elasticsearch. Connector management can simplify that arrangement without requiring ksqlDB to implement the search engine itself. The choice depends on the query the consuming application needs.

This is a useful limit on the consolidation argument. A supported point lookup can use the materialized state already maintained by the processor. A different application may still need a separate analytical or search store. Bringing connector management into the same interface can help operate that pipeline even when its final query destination remains another system.

The new name describes the interaction now offered, maintained tables, SQL processing and a network query interface. Its useful boundary remains specific. ksqlDB serves asynchronously materialized stream state and can manage the connectors supplying or exporting that stream. A pull query retrieves the supported current result, while a push query continues to report changes as processing advances.