Librdkafka 0.11.3 gives producer and consumer queues explicit byte defaults

A queue limit is meaningful only when its scope is known. A producer's total queued payload and a consumer's buffered data for one partition are different quantities, even if both settings have the same numerical default. Librdkafka 0.11.3 changes those defaults to one gigabyte. The versioned configuration reference shows why that does not create a one-gigabyte process-memory budget.

The 4 December maintenance release lists both producer and consumer queue changes. The configuration reference at the release tag gives each default as 1048576 kilobytes. It describes their scopes separately, which is the important operational detail behind the shared number.

I value Kafka's ability to let consumers advance through retained history at their own pace. Client queues support that independence by decoupling broker fetches from application processing. Their limits still need to reflect the application's actual memory budget and number of assigned partitions.

The producer limit measures queued message contents

The producer property queue.buffering.max.kbytes limits the sum of message sizes allowed on its queue. The reference gives that byte-oriented property higher priority than queue.buffering.max.messages. Message count and payload size constrain different things.

A count limit alone is insufficient when record sizes vary. A queue containing the same number of messages can hold very different quantities of payload. The byte limit gives the client a separate way to constrain that variation without assuming one typical record size applies to every event.

The reference still defaults the message-count limit to 100000. The existence of the byte limit does not remove the count setting. An application can reach a configured constraint before the other quantity reaches its own limit, so interpreting either number in isolation is incomplete.

The documented byte quantity is the sum of message sizes. It is not a claim about every allocation associated with producer operation. The distinction leaves room for protocol buffers and other implementation state outside that payload accounting.

The consumer limit applies to each topic partition

The consumer property is queued.max.messages.kbytes. Its documentation in this release describes a maximum per topic and partition in the local consumer queue. That scope differs from the producer's total queue description.

Several assigned partitions can therefore each contribute buffered data. A person budgeting the client as though the setting were a single aggregate ceiling would underestimate the scope described by this version's documentation. The relevant input is not merely the default number, but the number of partition queues involved.

The reference also says a fetch can overshoot the limit by fetch.message.max.bytes. The queue boundary is consequently not an exact instantaneous memory ceiling even for one partition. Fetching a unit of data and subsequently accounting for the queue can temporarily exceed the nominal threshold.

A configuration fragment can make the separate choices visible:

queue.buffering.max.kbytes=65536
queued.max.messages.kbytes=8192

These are illustrative explicit limits, not a measured recommendation for an application. The first applies to producer queue contents. The second uses the consumer scope documented for this release. A deployment using only one client role needs the properties relevant to that role.

Fetch size has a different responsibility

The initial per-partition fetch size defaults to 1048576 bytes in this version. If a record exceeds that size, the client can increase the request allowance until it can fetch the complete message. This prevents an initial request size from permanently blocking a larger record.

The maximum protocol response size is another setting. Its documentation describes a safeguard against memory exhaustion from malformed or unexpected protocol responses. It has a different job from controlling how much already fetched data waits for the application.

The consumer also has a desired minimum message count for local queues. The byte limit takes precedence over that count target. This lets a memory constraint stop further accumulation even when the queue has not reached the preferred number of messages.

These distinctions matter when a consumer is catching up. Retained history can provide a large backlog, while the application processes at its own rate. The client needs buffering for efficient transfer without interpreting the existence of backlog as permission to allocate memory indefinitely.

Windows authentication corrects an identity distinction

The same release changes Windows Kerberos handling to use sasl.kerberos.service.name for the broker principal instead of sasl.kerberos.principal. The latter describes the client's identity, while the former names the service the client is contacting.

The versioned reference says Windows uses the logged-in user's principal and does not support selecting the client principal through that property. That platform qualification explains why treating the two configuration names as interchangeable leads to incorrect behavior.

This is separate from queue sizing. Both changes demonstrate why defaults and property descriptions deserve version-specific reading, since familiar names can retain distinct roles across platform and client paths.

I prefer explicit queue settings when memory is constrained because the intended scope remains visible after a library upgrade. Librdkafka 0.11.3 supplies new defaults, but the producer payload sum, each consumer partition queue and the protocol response allowance remain different quantities. The application's process budget has to account for their combined use alongside the rest of its own memory.