Confluent Platform 3.2 brings the newer Kafka consumer to HTTP

An HTTP interface to Kafka still needs to represent a consumer's position and group membership. Confluent Platform 3.2 makes that relationship more explicit through the REST Proxy's v2 consumer API. Creating a consumer, subscribing it to a topic and fetching its records are separate operations. The client uses HTTP, but the proxy maintains a Kafka consumer on its behalf.

The final 3.2.0 release tag is dated March 2, 2017. Its accompanying Schema Registry build specifies Kafka 0.10.2.0-cp1. The distribution therefore brings that Kafka release together with the surrounding services. For applications accessing Kafka over HTTP, the REST consumer interface is a concrete part of what this package changes.

I value Kafka's ability to let several systems consume the same durable history at their own pace. An HTTP interface is useful when it makes that model accessible to another application language. Its consumer lifecycle still needs to be visible enough for the application to manage its position and resources.

A subscription becomes an explicit operation

The 3.1.0 quickstart creates a consumer and then reads from a URL ending in /topics/jsontest. Its documentation describes the high-level Kafka consumer behind that interface. The 3.2.0 introduction distinguishes the two implementations. API v1 uses the high-level consumer, while v2 uses the newer consumer introduced with Kafka 0.9.

That distinction changes the request sequence. The v2 example creates an instance within a consumer group, posts a topic subscription to that instance and reads through its /records endpoint. Subscription is now an operation on the consumer resource. The example also changes the offset reset value from the older smallest spelling to earliest.

The versioned quickstart gives the following sequence for a local proxy with a topic named jsontest. This assumes the proxy and Kafka are already running and the topic contains JSON records.

curl -X POST \
  -H 'Content-Type: application/vnd.kafka.v2+json' \
  --data '{"name":"reader","format":"json","auto.offset.reset":"earliest"}' \
  http://localhost:8082/consumers/reporting

The response identifies the consumer instance and returns its base_uri. In this local example, that URI is http://localhost:8082/consumers/reporting/instances/reader. Subsequent requests use that address.

curl -X POST \
  -H 'Content-Type: application/vnd.kafka.v2+json' \
  --data '{"topics":["jsontest"]}' \
  http://localhost:8082/consumers/reporting/instances/reader/subscription

curl \
  -H 'Accept: application/vnd.kafka.json.v2+json' \
  http://localhost:8082/consumers/reporting/instances/reader/records

The separate subscription request gives the HTTP client a direct way to express which topic it wants this consumer to read. Record fetching then addresses the consumer's records resource. Applications adapting the older quickstart need to change this request sequence as well as the version in their headers.

Version and data format are separate choices

The tagged API reference describes media types that encode the API version and the embedded record format. In application/vnd.kafka.json.v2+json, the embedded format is JSON, the interface version is v2 and the request or response envelope also uses JSON. Consumer creation and subscription use application/vnd.kafka.v2+json because those requests do not embed Kafka records.

This distinction also explains the binary interface. A JSON HTTP envelope can carry binary record data as base64 strings. Avro requests use JSON representations of the data with a schema or schema identifier. The proxy handles serialization before writing to Kafka, and its Avro support uses Schema Registry. Choosing JSON as an HTTP envelope does not make all the stored record formats interchangeable.

The 3.2 documentation supports both v1 and v2. Explicit media types let an application select its interface deliberately. A client that only changes a URL while retaining v1 headers has not made the same request as the v2 quickstart. The headers belong in the application's interface contract alongside its endpoint paths.

Consumer state determines request routing

The release's introduction states that consumers belong to specific REST Proxy instances. The returned base_uri identifies where later requests must go. A load balancer can distribute initial requests across proxies, but existing consumers require requests to reach the process that owns them.

The deployment documentation also explains that restarting a proxy with active group members causes a consumer rebalance. A deployment therefore needs to account for those rebalances when restarting several proxies. There is no persistent proxy state to migrate, but active consumer instances still affect the behaviour of a rolling update.

This is the operational cost I associate with the HTTP interface. It removes the application's need to speak the Kafka protocol directly, while adding a service that owns live consumers. Routing and process lifetime become part of the application's consumption behaviour.

The quickstart closes its consumer explicitly after reading. For the example above, cleanup uses the same instance address.

curl -X DELETE \
  -H 'Content-Type: application/vnd.kafka.v2+json' \
  http://localhost:8082/consumers/reporting/instances/reader

The proxy shares producer resources

The producer side has a different resource model. The 3.2 introduction describes a small shared pool of producers behind topic and partition requests. Producer configuration applies globally through proxy settings, so individual requests cannot select arbitrary producer options. Consumer instances are separate, although the API exposes only limited configuration because they share server resources.

That difference matters when several applications use one proxy. A global producer setting such as compression affects the proxy's producers, while a named consumer has its own lifecycle and position. Capacity planning must account for the number of live consumers as well as HTTP traffic. The configuration includes an idle consumer timeout, and the quickstart's explicit deletion gives applications a way to release resources when they finish.

Confluent Platform 3.2 makes the newer Kafka consumer available through a versioned HTTP interface. An application adopting it needs the v2 media types and the subscription request sequence. Its deployment must also preserve access to the consumer's owning proxy for the lifetime of that instance.