librdkafka 1.1.0 adds broker identity checks and OAuth callbacks

An encrypted Kafka connection still needs to establish which broker is at the other end. The librdkafka 1.1.0 release on July 5 makes that distinction visible in its client configuration. OpenSSL broker certificate verification is enabled by default, while the newly available hostname verification remains disabled by default. Applications need to understand both settings before treating a successful TLS handshake as sufficient evidence of broker identity.

The release also adds SASL OAUTHBEARER support and ways to supply certificates from memory. Those changes address different parts of connection establishment. Certificate verification concerns the remote peer, client certificates can identify the connecting application, and the OAuth API lets application code supply a token for SASL authentication. Their configuration and execution paths remain separate.

I value Kafka for the durable history it makes available to independent consumers. That makes client identity consequential. A system that treats records as reusable history needs deliberate rules for the connections through which applications access that history. This release is useful because it exposes several of those rules in the C client API, where applications can configure and inspect them.

Certificate trust and hostname checks have separate controls

The configuration reference at the 1.1.0 tag gives enable.ssl.certificate.verification a default of true. It describes this as OpenSSL's built-in verification of the broker certificate. The same reference gives ssl.endpoint.identification.algorithm a default of none, with https as the alternative. Selecting https enables broker hostname verification according to RFC 2818 and requires OpenSSL 1.0.2 or newer.

The distinction is operationally concrete. Trusting the authority behind a certificate does not by itself establish that the certificate names the broker the client intended to contact. Hostname verification supplies that additional comparison. A configuration that preserves certificate verification while leaving endpoint identification at none has not requested the hostname check described by the new setting.

The configuration values below are an illustrative TLS fragment for librdkafka 1.1.0. They specify the transport and make both verification choices explicit. An application still supplies its broker addresses and the appropriate trust material for its environment through the rest of its configuration.

security.protocol=ssl
enable.ssl.certificate.verification=true
ssl.endpoint.identification.algorithm=https

The spelling matters. One section of the release notes lists enable.ssl.endpoint.identification.algorithm, but the tagged configuration reference uses ssl.endpoint.identification.algorithm. The earlier summary in those same release notes uses the latter spelling too. The tagged reference is the appropriate source for the property name in an application configuration.

The TLS implementation at this tag shows the separation in code. Its endpoint-identification setup returns without enabling hostname verification when the configured algorithm is none. For supported OpenSSL versions, the other path configures the expected hostname through the corresponding OpenSSL API. This provides a concrete explanation of why enabling certificate verification does not silently enable the separate hostname option.

Application certificate policy can extend the built-in check

The public header shipped with 1.1.0 documents rd_kafka_conf_set_ssl_cert_verify_cb. The callback receives the broker name and identifier, the certificate in DER form, and the result of OpenSSL verification. It runs for certificates in the broker chain and returns whether verification succeeds. The application can also supply a human-readable error when rejecting a certificate.

This is a policy extension with a specific execution context. The header says the callback runs on internal librdkafka threads. Application code using it therefore has to account for that thread context when accessing its own state. A callback integrated with application trust policy is materially different from a configuration value read once during startup.

The header also allows the callback to clear an existing OpenSSL verification error by returning success and resetting the error code. That makes the callback capable of changing the effective verification result. Calling it an additional check alone would conceal that capability. Its implementation needs to define when it accepts the original result and when it deliberately substitutes an application decision.

I prefer keeping ordinary certificate and hostname verification explicit even when an application needs a callback. It makes the baseline policy readable in configuration and leaves the callback responsible for a named additional requirement. The API provides the mechanism, but the application supplies the acceptance rule and any state needed to evaluate it.

Credentials no longer have to arrive through files

The same header documents rd_kafka_conf_set_ssl_cert for supplying certificates or keys through an encoded memory buffer. Callers identify the material as a public key, private key, or CA certificate and specify its encoding. The API recognizes PEM, DER and PKCS#12 encodings. A failed decode or a build without SSL support produces a configuration error.

The buffer lifetime is explicit. The function does not retain a reference to the supplied memory after returning. Repeating the call for the same certificate type replaces the previous configured value, and a null buffer clears that configuration. Where a private key requires a password, ssl.key.password must be configured before this function runs.

For PEM material, the configuration reference also offers ssl.key.pem and ssl.certificate.pem string properties. These complement the existing file-location settings. An application that already obtains credential bytes can pass them into the configuration without first inventing a file path solely for the client library. That is a practical integration choice, not a statement that holding secrets in process memory removes their sensitivity.

The Windows trust behavior changes as well. The release notes say librdkafka loads CA certificates from the Windows Root Certificate Store by default, so Windows SSL users no longer need to specify a CA file or directory through ssl.ca.location. This changes where trust material comes from. It does not change the separate default for hostname verification.

OAuth requires an application-managed token lifecycle

SASL OAUTHBEARER support adds a token mechanism to the client, with a refresh callback through which application code obtains credentials. The tagged configuration reference accepts OAUTHBEARER as a sasl.mechanisms value. Despite that property's plural spelling, the reference says to configure only one SASL mechanism. Transport selection remains a separate security.protocol setting, with sasl_ssl available for SASL over TLS.

The public header documents rd_kafka_conf_set_oauthbearer_token_refresh_cb as the entry point for registering the refresh handler. Polling triggers the callback when a token needs retrieval. Application code then reports success through rd_kafka_oauthbearer_set_token or failure through rd_kafka_oauthbearer_set_token_failure. A queue-based application can receive the refresh request as an RD_KAFKA_EVENT_OAUTHBEARER_TOKEN_REFRESH event instead.

That initial token is required before a SASL OAUTHBEARER broker connection can succeed. The header explicitly describes supplying it directly or invoking polling so that retrieval occurs. A program that configures the mechanism but never services the refresh path has omitted a required part of connection establishment. Token retrieval is part of the client lifecycle, including startup.

The token setter accepts more than the token text. Its parameters include the Kafka principal name and an absolute expiration timestamp in milliseconds since the epoch. That last detail matters because a duration such as an hour is not the value the API requests. The function also accepts optional SASL extension key-value pairs and documents restrictions on their format, including the reserved auth key.

These interfaces leave token acquisition with application code. They do not prescribe an identity-provider endpoint or automatically select a grant flow. The application can use the refresh callback to connect its credential source to librdkafka, then report a token with the required metadata. Treating support for OAUTHBEARER as a complete identity-provider integration would skip that work.

The built-in unsecured handler has a limited purpose

The configuration reference provides enable.sasl.oauthbearer.unsecure.jwt, disabled by default. It enables a built-in unsecured JWT handler when no refresh callback has been set. The documentation explicitly limits that handler to development and testing. The header also says that using it and registering a refresh callback are mutually exclusive choices.

Its configuration illustrates why the two paths must remain distinguishable. The built-in handler accepts settings such as a principal and token lifetime through sasl.oauthbearer.config. Supplying those values can help exercise the authentication mechanism during development. It does not establish the external credential acquisition and validation policy an application needs for deployment.

For application-managed credentials, the refresh API provides a concrete failure path. rd_kafka_oauthbearer_set_token_failure requires a human-readable reason when acquisition fails. The token setter can separately reject invalid arguments or report that the build lacks support or the client has not configured OAUTHBEARER. Those outcomes distinguish a credential retrieval failure from a client configuration error, which gives application code specific information to handle during connection startup and later refreshes.