Kafka 0.10.2 adds SCRAM credentials stored in ZooKeeper

Password authentication has a storage problem as well as a connection problem. A broker needs enough information to verify a client without making its credential database a collection of reusable plaintext passwords. Kafka 0.10.2 adds SCRAM authentication, with a stored identity derived from the password and a separate requirement to protect the authentication exchange through TLS.

The Apache release entry dates 0.10.2.0 to 21 February. The release notes identify SASL/SCRAM support as KAFKA-3751. They also introduce per-client JAAS configuration, which makes the choice of credentials less dependent on a single process-wide configuration file.

I value Kafka as retained history that several systems can consume independently. Those systems need independent identities too. Adding a password mechanism is useful when its storage and connection requirements are clear enough to operate without confusing authentication with permission to read every topic.

SCRAM stores a derived identity

The security documentation at 0.10.2.0 supports SCRAM-SHA-256 and SCRAM-SHA-512. It describes the default credential store as ZooKeeper. Creating credentials produces a random salt and stores the salt, iteration count, StoredKey and ServerKey there.

Those stored fields differ from a plaintext-password database. The documented mechanism follows RFC 5802, with password-derived material used in the authentication exchange. The choice of iteration count affects the work involved in deriving that material. Kafka requires at least 4096 iterations and uses that value by default when none is specified.

The configuration tool can create entries for both mechanisms under one user. The documentation's example supplies a higher iteration count for SHA-256 while leaving SHA-512 at its default. Mechanisms therefore have distinct credential entries even when an administrator uses the same input password for both.

ZooKeeper remains a substantive dependency. The documentation recommends the default implementation for installations where ZooKeeper is secure and on a private network. Storing derived credentials changes what the store contains, but does not make unauthorized access to that store harmless.

Credentials have a creation and rotation lifecycle

Kafka uses kafka-configs.sh to create and manage the SCRAM entries. Credentials for broker-to-broker communication must exist before the brokers start. Client credentials can be created and updated dynamically, and new connections use the updated credentials.

The new-connection qualification matters during credential replacement. Updating the store is not described as retroactively reauthenticating every existing connection. A deployment needs to distinguish changing the authentication material from establishing new sessions that use it.

The same tool can describe user credentials or delete one mechanism's entry. This lets an administrator manage the enabled identities without treating every change as editing each broker's JAAS file. The credential store and the broker's own outbound login configuration have different jobs.

A broker's JAAS username and password are the credentials it uses when initiating connections to other brokers. The documentation's server example uses a dedicated administrative identity for this relationship. Client users still authenticate with their own supplied identities, which become principals for access-control configuration.

TLS protects the exchange separately

The documentation says SCRAM should be used with TLS encryption to prevent interception of the exchange. The stored credential representation and the transport protection address different parts of the authentication system. Selecting SCRAM does not by itself encrypt application traffic.

The client configuration names both the security protocol and the SASL mechanism. A fragment can make that separation visible without including a password:

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-256

This is only part of a client configuration. Trust material and JAAS credentials still need appropriate configuration for the target cluster. The first line selects the protected transport with SASL, while the second chooses the authentication mechanism used on that transport.

The release documentation also states that hostname verification is not enabled by default. Its SSL instructions describe enabling it with ssl.endpoint.identification.algorithm=HTTPS. This is a version-specific behavior worth making explicit because merely trusting a certificate issuer and verifying the expected server name are different checks.

Each client can have its own JAAS configuration

The release adds sasl.jaas.config as a client property. The security guide says this can replace a physical JAAS configuration file for a producer or consumer. Different client objects in the same JVM can therefore provide different credentials.

That is a useful change for a process serving several Kafka relationships. A process-wide JAAS file gives all client connections the same selected login section. A per-client property lets the application construct one client for one identity and another client for a different identity without splitting them solely to separate credential configuration.

The property takes precedence if the JVM's static JAAS configuration and the client property are both supplied. This explicit rule matters during a migration. An application should not assume the old file continues to determine credentials once a client-specific login configuration is present.

I prefer this allocation because a consumer's configuration can describe the identity associated with that consumer. The application still has to obtain and protect the password, but the client interface no longer forces every connection in the process to share one static credential choice.

Authentication leaves authorization policy intact

The guide states that the authenticated username becomes the principal used for ACLs. Authenticating a connection therefore identifies the caller. The authorizer separately decides which operations that principal may perform against topics, groups or other resources.

This distinction is especially relevant to replay. The retained log may contain more history than a particular application needs to consume. An authentication mechanism can establish who the client is without deciding the scope of history it may read. That remains an access policy expressed through the authorization system.

Brokers can enable several SASL mechanisms during a transition. The documented migration sequence enables the new mechanism on brokers, changes clients and then changes broker-to-broker authentication if required. Removing the old mechanism is a later step. The ordering allows the cluster to accept both populations while clients move.

Kafka 0.10.2 therefore adds more than a mechanism name. It defines where SCRAM identities live, how new connections use updated credentials and how an individual client supplies its login. The resulting authenticated principal enters the existing authorization model, while TLS and ZooKeeper protection remain explicit parts of the deployment.