librdkafka 1.9 adds OIDC login and native ACL administration

Kafka authentication capabilities need corresponding implementations in clients outside the Java ecosystem. Librdkafka 1.9 adds OIDC support for SASL/OAUTHBEARER and administrative ACL APIs. The June 16 release identifies these as implementations of KIP-768 and KIP-140. Native applications gain a built-in provider login path alongside a way to manage authorization entries.

The distinction between those capabilities matters. OIDC obtains a token that establishes a client's identity. ACL administration changes permissions stored by the cluster. An application can need the first without being authorized to use the second.

I value Kafka's retained history, but access to it should have an identifiable authority. The native client's provider integration is useful because it makes token acquisition an explicit part of client configuration, without requiring every application to supply its own token retrieval callback.

OIDC is an explicitly selected login method

The 1.9 configuration reference introduces sasl.oauthbearer.method. Its default value is default, and selecting oidc requires the client ID, client secret and token endpoint URL. Merely choosing the OAUTHBEARER SASL mechanism does not supply those provider-specific details.

The relevant configuration can be illustrated without putting a real secret into an example.

sasl.mechanism=OAUTHBEARER
sasl.oauthbearer.method=oidc
sasl.oauthbearer.client.id=example-service
sasl.oauthbearer.client.secret=replace-with-provider-secret
sasl.oauthbearer.token.endpoint.url=https://identity.example.com/oauth2/token

This is a partial client configuration. The application's broker connection and transport settings are separate. That endpoint belongs to the identity provider, and the credentials identify the application there. The token obtained through that endpoint is then used for Kafka authentication.

The reference also includes optional scope and extension properties. It separately documents an unsecured JWT handler for development and testing. That handler is not the new provider integration. Keeping those choices distinct prevents an example using a local test token from being mistaken for a deployment that validates provider-issued identity.

Callback scheduling can move to a background thread

The versioned changelog describes another OAuth improvement. Custom refresh callbacks previously depended on the application calling polling functions at least once before the callback could run and connections could proceed. The release adds APIs for enabling a SASL queue and background callback execution.

This addresses the lifecycle around authentication. An application can have a valid callback implementation while failing to schedule the work needed to obtain its first token. Moving that work to the library's background thread supplies another execution arrangement. The application still needs to choose the callback model its code supports.

That difference is useful when describing a client that cannot connect. The failure can concern callback execution before the broker exchange begins. Correct provider credentials alone do not establish that the application has triggered the code that uses them.

Connection setup receives a time boundary

The release introduces socket.connection.setup.timeout.ms with a 30-second default. It covers TCP connection setup as well as SSL and SASL handshakes. The changelog connects the limit to connections that stall during network or load-balancer problems. This gives connection establishment an explicit deadline instead of assuming a successful TCP exchange completes the whole setup. It is a separate timeout from the application operation performed after authentication.

ACL bindings describe permissions separately

The public C header defines ACL binding objects with resource type, resource name and pattern, alongside principal, host, operation and permission type. A binding can be passed to rd_kafka_CreateAcls. Filters support describing and deleting entries through separate operations.

These objects make the permission request explicit. Authentication identifies a principal, while the binding describes an operation that principal may perform against a resource. The client library now has an administrative representation for that request. It does not automatically grant an authenticated application permission to administer ACLs.

My preference is to keep those roles separate in application design. A service consuming history and a process changing access policy have different responsibilities, even if both use the same native client library.

Offset storage now checks partition ownership

The release also changes rd_kafka_offsets_store and related operations. Storing an offset for a partition not currently assigned returns a per-partition state error. The changelog explains the race this prevents. An application could store progress after revocation, then commit that old stored offset if the partition returned later, overwriting progress committed by another consumer in the meantime.

The library additionally clears stored offsets on assignment. Applications using explicit offset storage must handle RD_KAFKA_RESP_ERR__STATE in the partition result. That error says the consumer no longer owns the partition for the requested storage operation.

This is consequential for Kafka's independent consumer positions. Ownership changes must not let a stale local offset become current progress again. Alongside the identity and ACL additions, the release makes the client more explicit about which authority it has, both when connecting to a broker and when storing progress for a partition.