Confluent BigQuery Sink V2 uses the Storage Write API

A Kafka record's arrival at a warehouse involves a second write protocol with its own acknowledgements and failure handling. Confluent's November 17 release note makes the BigQuery Sink V2 connector available for Kafka clusters on Google Cloud. The fully managed connector uses Google's BigQuery Storage Write API. Understanding that API explains the architectural change more precisely than treating V2 as an interchangeable connector label.

Google introduced the API before this connector release. Its May 2021 overview describes stream-level transactions and schema-update detection. The November connector announcement makes that ingestion mechanism available through a managed Kafka sink. It also says the API may provide a cost benefit. It does not establish a universal cost reduction for every workload or describe an end-to-end delivery guarantee merely by naming the API.

I value Kafka because a retained stream can feed independent systems. A warehouse sink is one such consumer, with a destination that has its own rules for accepting data. The useful question is how that consumer translates progress in a partition log into progress at the destination. A new write API changes the available mechanisms for that translation.

A write stream belongs to a destination table

Google's July 2023 API message definitions describe AppendRows as a bidirectional streaming RPC. A request identifies a BigQuery write stream. Its resource name includes the project, dataset and table, followed by the stream identifier. A special default stream uses _default in that final position. This is a destination-side stream, with a namespace unrelated to Kafka's topic partitions.

Append requests can carry protocol-buffer rows and the writer's schema. The request documentation requires a schema for the first request on a connection and when the input schema changes. The server's response can report an updated table schema. That exchange provides a concrete mechanism for dealing with a destination schema that changes during a writer's lifetime. It does not by itself define how a Kafka connector maps each source field to a warehouse column.

The API also defines an optional append offset. When supplied, the write proceeds only if the stream's next append position matches the supplied value. A duplicate position produces ALREADY_EXISTS, while a position beyond the current end produces OUT_OF_RANGE. These responses give an API client information about repeated or incorrectly positioned requests. The default stream does not allow this offset field.

That last qualification prevents a misleading inference. Support for an API that offers offset checks does not prove that every user of the API uses those checks. A claim about connector delivery must describe the connector's chosen stream and recovery behavior. The release note establishes the ingestion API, but it cannot substitute for that fuller contract.

Visibility depends on stream behavior

The matching write-stream definitions distinguish committed, pending and buffered streams. Committed data appears when the write is acknowledged. Pending data remains invisible until commit. Buffered data is visible only through the position that has been flushed. These are API capabilities, rather than a claim that the November managed connector exposes every stream type as a configuration option.

The distinction explains why an ingestion API can support both continuous and batch-oriented work. Sending rows and making rows visible can be separate actions. The API's BatchCommitWriteStreams request identifies a group of streams belonging to one parent table and commits them atomically. Its response reports stream errors, and the documented atomicity means that an unsuccessful group commit does not commit a subset of that group.

That scope matters when comparing destination transactions with a Kafka pipeline. An atomic commit of BigQuery write streams is a statement about those destination streams. It does not automatically include an unrelated Kafka consumer offset commit. The pipeline must connect its source progress and its destination acknowledgements through the connector's recovery logic. Both systems can have precise local contracts without those contracts forming one shared transaction.

I regard that distinction as part of designing for replay. Reading the same Kafka records again is possible while retention permits it. Whether another delivery produces duplicate warehouse rows depends on the sink's behavior and destination operations. Source replay and destination deduplication need separate explanations, even when a destination API has useful position checks.

Two positions describe different progress

Consider a sink that has read a record but loses its connection before it can establish the outcome of a destination write. A retry must resolve what BigQuery accepted. A restart must also determine which Kafka records still need delivery. This is an analytical example of the two progress records involved, not a description of an undocumented implementation inside Confluent's managed connector.

The API definitions make some of the relevant evidence explicit. Append errors indicate rows were not accepted and distinguish malformed requests from retryable server failures. An append offset can identify an already accepted position when that mechanism is in use. Schema feedback describes another reason that a request may need to change before retrying. A sink has to interpret these outcomes in the context of the Kafka records it is delivering.

For the November release, the confirmed change is a managed ingestion path using this API for Kafka clusters on Google Cloud. Its practical significance is the use of a destination protocol that separates stream identity, append position and visibility. Those details give engineers concrete questions about a warehouse sink's recovery contract while keeping Kafka's retained history available to its other consumers.