Strimzi packages Kafka's stateful deployment requirements

Putting Kafka in a container does not decide what happens to a broker's identity and data when that container is replaced. Strimzi's first release on January 16 packages those decisions into Docker images, OpenShift templates and Kubernetes YAML. The release is useful because its deployment definitions describe more than how to start the broker process.

The README at the 0.1.0 tag distinguishes a StatefulSet deployment from a separate ephemeral configuration. It explains how broker identities relate to pod names and networking, and how each broker obtains its own persistent volume. That is a concrete starting point for running a distributed log on a container scheduler.

I regard Kafka's retained log and independent consumer positions as its most useful architectural properties. Preserving that history requires attention to the storage beneath the broker. A deployment file that makes the storage lifetime explicit is therefore more informative than a container image that only demonstrates a successful startup.

Identity and storage need corresponding lifetimes

The StatefulSet configuration gives pods unique names and network identities. Strimzi uses those identities to help identify individual brokers and assign broker IDs. Each Kafka broker pod has its own persistent volume, obtained through a PersistentVolumeClaim. The claim separates the deployment definition from the concrete kind of storage satisfying it.

The README gives local HostPath storage and Amazon EBS as examples of backing storage that can satisfy this arrangement without rewriting the Kafka deployment YAML. That is a claim about the abstraction used by the definitions. It does not establish that those storage systems have interchangeable performance or failure behavior. The same claim structure can refer to different implementations while the operator remains responsible for the implementation's properties.

Strimzi also distinguishes regular services from headless services. The former provide bootstrap endpoints for clients. Headless services let DNS resolve pod IP addresses directly. The deployment consequently expresses both a way to enter the Kafka cluster and a way to address its individual members.

This distinction follows the identity requirements in the deployment. A single bootstrap endpoint is convenient for discovering the cluster, but the individual broker pods still need identifiable addresses. The YAML contains both service forms because they answer different addressing questions. Replacing one with the other would discard part of the intended arrangement.

The OpenShift template described by the README creates a three-replica ZooKeeper StatefulSet and a three-replica Kafka StatefulSet. It also creates the regular and headless services for each. These are the supplied template's defaults, not a universal statement about how many replicas every Kafka deployment should use.

The ephemeral configuration changes the data contract

Strimzi's alternative deployment uses emptyDir instead of persistent volume claims for ZooKeeper information and Kafka topic and partition storage. The README explicitly limits this configuration to development and testing. Its contents follow the pod's lifetime, so replacing the pod does not provide the same preservation as a persistent volume arrangement.

The distinction is easy to miss if both deployments are assessed only by whether a client can produce and consume a message. Both can demonstrate the application protocol. They provide different answers when the storage-owning pod disappears. For a retained log, that answer is part of the service behavior being deployed.

I find the explicit development configuration helpful because it reduces ambiguity. A disposable environment can avoid provisioning persistent volumes while still exercising client integration. The same convenience should not be interpreted as evidence that the persistent configuration is unnecessary for workloads whose history must survive pod replacement.

Connect adds its own packaging problem

The release includes a Kafka Connect deployment with a configurable worker count. Its initial image contains the file source and file sink connectors distributed with Apache Kafka. The README explains that additional plugins belong under /opt/kafka/plugins and describes several ways to supply them.

One method builds a derived image containing the required connector files. Another mounts a volume with the plugins, while OpenShift users can use the supplied Source-to-Image arrangement. The choice determines where the plugin installation becomes part of the deployed environment. A derived image records it in an artifact, whereas a mounted directory introduces a separate dependency on available volume contents.

The documentation also notes the multiple-worker constraint for mounted plugin storage. A volume used by workers on different nodes must be readable from those pods. Merely identifying a directory on one machine cannot establish that every worker sees the same plugin files. This is a deployment dependency separate from reaching the Kafka bootstrap endpoint.

Metrics include the broker and its JVM

Each Kafka broker and ZooKeeper pod exposes a Prometheus endpoint through a JMX exporter running as a Java agent. The README describes both JVM metrics and broker-specific metrics. This makes the runtime process observable alongside the Kafka behavior it implements.

The January release therefore provides several inspectable parts of a deployment. Broker identity has a StatefulSet representation, durable storage has claims, and temporary storage has an explicitly different lifetime. Connect plugins have a documented installation location, while the metrics endpoint exposes information about the broker process. These definitions let someone examine what the cluster needs before treating a running pod as evidence that the log's operational requirements have been satisfied.