librdkafka 1.5 makes unavailable topics visible to consumers
The July release reports subscription failures explicitly and gives topic creation time to propagate. It also adds a batch-size control and changes producer accumulation defaults.
An empty consumer result can mean that no records have arrived, but it can also conceal a topic that is absent or inaccessible. librdkafka 1.5.0 changes how those cases reach the application. The 20 July release propagates subscription errors for unavailable topics and improves broker-connection diagnostics. It also changes topic-creation behavior and adds a producer batch-size setting.
These are useful changes because the client has information that an application cannot infer from a quiet input stream. An explicit authorization failure describes a different condition from a valid subscription with no new records. The application needs that distinction before it can interpret a lack of consumption as ordinary inactivity.
I value Kafka's retained history partly because a consumer controls its position independently. That control assumes the consumer can establish which topic it is reading and whether it has access. Better reporting at that boundary helps preserve the distinction between an idle application and one that cannot read its intended history.
Missing and unauthorized topics produce errors
The tagged changelog identifies UNKNOWN_TOPIC_OR_PART and TOPIC_AUTHORIZATION_FAILED as errors now delivered through the standard consumer message error field. Applications therefore need to interpret an error-bearing result as part of consumption. A poll result is not necessarily a business record, and its error field can explain why the expected record stream is unavailable.
The release also stops consumers from automatically creating missing topics by default. The 1.5.0 configuration reference documents allow.auto.create.topics as false. Setting it true requires the broker's automatic-creation setting to permit the operation too. The reference explicitly notes that this default differs from the Java consumer's default.
That last detail is consequential for applications using several client implementations. The same missing topic name need not produce the same creation behavior from each client's defaults. Topic provisioning becomes an explicit part of the environment when the consumer no longer attempts to create the topic merely by subscribing. An authorization failure also remains different from a missing topic, even if neither yields data.
Recent creation is different from permanent absence
The release adds topic.metadata.propagation.max.ms because topic creation is asynchronous. Its configuration reference describes a client asking one broker for metadata before a manually created topic has propagated through the cluster. The broker can make the topic appear absent even though its creation is already underway. Immediately treating that observation as permanent can fail queued producer messages.
The new setting delays marking the topic nonexistent. Its default is 30,000 milliseconds, measured from the first reference to the topic in the client, such as a produce call. This is a bounded allowance for propagation. It is not a guarantee that an arbitrary misspelled topic will become valid or that authorization problems disappear with time.
The combination is useful. Consumers stop creating topics implicitly, while clients give an explicitly created topic time to become visible. Those changes distinguish who creates a resource from how quickly all metadata readers learn about it. More explicit errors remain compatible with allowing for a known asynchronous transition.
Batch size and accumulation time control different dimensions
Producer batch.size now limits the bytes in a MessageSet, including protocol framing overhead. The reference says the limit applies after adding the first message, so a first message larger than that setting is not excluded solely by batch.size. Other limits, including message.max.bytes and the maximum number of batched messages, still apply.
This makes batch.size different from a universal maximum message size. It controls how messages accumulate into a batch, while the first-message rule permits forward progress for a message that is otherwise acceptable. The distinction matters when reasoning about why one batch can exceed the configured accumulation target.
The release also increases default linger.ms from 0.5 to 5 milliseconds. Its configuration description explains the tradeoff. Waiting allows more messages to accumulate, improving batching and compression at the cost of additional delivery latency. An application needing a lower accumulation delay can configure a smaller value. This is a changed default policy, separate from adding the byte-oriented control.
I prefer those explicit dimensions to a single throughput recommendation. A byte limit constrains batch composition. An accumulation interval controls time spent waiting for more records. The application chooses both in the context of its record sizes and latency requirements.
The July release consequently changes how a client reports an invalid input stream and how a producer accumulates valid output. Missing-topic errors now reach the application, recently created topics receive a defined propagation interval, and batching has clearer controls. Each change exposes a decision that was previously harder to distinguish from ordinary client inactivity or default behavior.