Confluent goes public around the problem of continuous data integration

The problem described in Confluent's IPO letter is connecting applications that need to react to one another continuously. Jay Kreps writes on June 24 that Confluent has become a publicly traded company. His explanation of the business begins with the difficulty of uniting different applications and data stores, rather than with a claim that companies lacked another place to store data.

That distinction is the useful technical subject behind the commercial milestone. A database can preserve an application's current state while other applications still need a reliable account of the changes that produced it. A stream gives consumers something they can process independently, at their own position in that history. The integration problem includes how those consumers obtain and interpret it.

I find Kafka's retained log more interesting than treating it as another queue. Independent consumption and replay change how a system can introduce another reader of existing data. They also leave design choices, particularly partitioning, that a vendor's success cannot decide for an application.

The letter describes an integration problem

Kreps attributes Kafka's origin to work at LinkedIn with Jun Rao and Neha Narkhede, who later became Confluent's co-founders. He describes needing applications and data stores to act as a coherent system. Existing storage technologies did not directly supply the continuous connection he wanted among those components.

The letter then describes Kafka's development as open source and its donation to the Apache Software Foundation. Confluent's business grew from encountering similar integration needs outside the original group of technology companies. These are claims made by a founder explaining the company's direction, rather than independent evidence that one architecture fits every organization.

The argument is still specific enough to examine. If several applications need to react to the same sequence of changes, connecting each producer separately to every consumer duplicates integration work. A shared stream creates a place where producers publish changes and readers obtain them. Whether that is appropriate depends on what the records mean and what consumers need to do with them.

The IPO does not change those application requirements. It is a financing and ownership milestone for the company building a commercial offering around them. The distinction matters because a software project's behavior should be assessed through its implementation and contracts, while a vendor's product also includes the services it supplies around that behavior.

The offering and the public debut have different dates

Confluent's pricing announcement describes 23 million Class A shares at $36 per share. It states that trading is expected to begin June 24 on the Nasdaq Global Select Market under CFLT. The expected closing date is June 28, subject to customary conditions.

Kreps's letter marks the public debut today. The expected later closing remains an announced future event. Keeping those descriptions separate avoids treating the start of public trading as evidence that every step of the offering has already finished.

For an engineer, the important commercial implication is that Confluent is articulating its product strategy to public investors as well as to customers. The letter presents continuous data integration as a long-term business opportunity. That is a statement of direction, not an additional guarantee about a Kafka deployment's delivery or ordering behavior.

Consumers choose a position in retained history

Kafka's 2.8 design documentation describes consumers issuing fetch requests to the brokers leading their partitions. A request supplies an offset, and the response contains records beginning at that position. A consumer can move its position backward to read earlier data again.

That makes the storage and consumption relationship different from deleting each message after one recipient acknowledges it. A consumer's progress is represented separately. The documentation explains that a partition is ordered and is assigned to one consumer within a subscribing group at a time. Position within that partition can be represented by the next offset to consume.

This is the mechanism that supports independent application progress. A reader that falls behind can later fetch more data without requiring the producer to hold that reader's connection open. Reprocessing can use an earlier position when the relevant history is still available. A new consumer can use the same stored records without consuming them on another application's behalf.

The pull model also lets consumers request available batches at their own pace. Kafka's design document contrasts that with a producer-driven downstream flow whose sender must decide how much a consumer can accept. Long polling avoids a tight request loop when no records are available. These choices concern the operation of the shared stream, not the financial structure of the company supplying related products.

A shared stream still requires an application contract

Offsets make progress representable, but they do not define the meaning of an application's records. A consumer needs to know what a change represents and how to apply it. Rewinding after a software defect can be useful, yet the downstream effects of processing the same record again still need an application-level policy.

The design documentation illustrates the reason by discussing acknowledgement failures. Processing can finish before an acknowledgement reaches the system tracking consumption. Repeating the work then has different consequences depending on the destination operation. An offset makes the input position compact and recoverable, but the application still needs to account for the relationship between that position and the effects it creates elsewhere.

Partition ordering likewise has a deliberate scope. The documented position is per partition, not a universal sequence across every topic and reader. An application using the log as shared history needs a partitioning decision that matches the ordering it requires. Moving more integration through a stream makes that decision more consequential.

I regard the IPO letter as a statement about the size of that integration problem. Kafka's design supplies concrete mechanisms for retained history and independent consumer progress. Confluent's commercial milestone concerns a company building around those mechanisms, while the application still has to define its record semantics and the scope of ordering it relies on.