NixOS 19.03 makes Kubernetes identity part of configuration

A Kubernetes cluster needs more than processes that can reach each other's ports. Its components need identities, and the API server needs rules governing what those identities may do. NixOS 19.03 refactors its Kubernetes modules around that requirement, enabling two-way TLS and role-based access control by default. The April 11 Koi announcement opens the stable channel, while the release-tag notes describe the module changes.

This is a consequential distribution change because enabling a service has to produce a coherent configuration across several components. Turning on authentication without giving clients appropriate credentials leaves a cluster unable to communicate. Preserving connectivity through insecure endpoints would defeat the new default. The module needs to coordinate both sides of the relationship.

The declarative treatment of systems is what attracts me to NixOS. This release makes that treatment more substantial for Kubernetes by including component identity and authorization in the configuration model. A declaration should describe the conditions under which the service can operate, including the credentials its peers need.

Roles select components without merging them into one service

The Kubernetes chapter shipped with 19.03 describes a collection of submodules. An administrator can enable individual components directly, or assign a host the master and node roles. These roles provide a convenient selection of components while retaining their separate configuration.

The master role enables the API server, controller manager, scheduler and addon manager, together with kube-proxy and etcd. The node role enables kubelet and kube-proxy. Assigning both is documented as useful for a single-node development or testing cluster.

A minimal fragment expressing that documented arrangement is straightforward.

services.kubernetes = {
  roles = [ "master" "node" ];
  masterAddress = "localhost";
};

This is a fragment for a NixOS configuration, not a complete machine installation. The localhost address is appropriate to the single-node case described in the manual. A cluster with additional machines needs an address those machines can resolve and route to.

The role assignment also defaults flannel and easyCerts to true. The manual describes these as CNI setup and automatic PKI bootstrapping. That is a useful example of a module coordinating related settings, but it also means that a role's effect extends beyond starting the most obvious daemon.

The cluster address becomes mandatory

NixOS 19.03 requires services.kubernetes.masterAddress. The manual says every cluster node must be able to resolve and route to it. This is a concrete dependency that can otherwise remain implicit in an example configuration.

A certificate-enabled distributed service needs its clients to agree about the endpoint they will use. Even before considering every certificate field, a master address that exists only in one machine's local view cannot provide a usable shared connection target. Making the setting mandatory exposes that part of cluster topology in the declaration.

The single-node localhost example is therefore a deliberately limited case. Copying it unchanged to a multi-machine configuration would preserve the spelling of a valid option while violating its documented reachability requirement. The type and presence of a setting cannot by themselves establish that the surrounding network makes its value meaningful.

That boundary is useful to understand when evaluating declarative configuration. NixOS can require the option and generate component settings from it. The administrator still supplies a name and network arrangement that satisfy the stated relationship among machines.

TLS identity and RBAC answer different questions

The release chapter says RBAC authorization is enabled by default. Anonymous requests to the API server's secure port should consequently receive a permission-denied response. All cluster components therefore need X.509 certificates for two-way TLS communication.

The certificate subject supplies identity information that the API server uses in assigning permissions. Encryption of the connection alone does not answer what the caller may do. The module's defaults combine authenticated communication with authorization, so a successful network connection is only part of the interaction.

This distinction changes what counts as a working cluster client. A component that previously relied on an insecure endpoint now needs credentials accepted by the secure endpoint. A client that has an identity still needs the permissions appropriate to its operations. Treating TLS and RBAC as a single switch would hide those separate requirements.

The release notes describe component decoupling and security as goals of the refactoring. Separate components can retain their own configuration while participating in the same identity system. That arrangement makes the relationships explicit without requiring every process to run with an undifferentiated shared level of access.

Certificate bootstrapping is a running service

The easyCerts option supplies automatic certificate bootstrapping and configuration. According to the period manual, it sets up a cfssl certificate-authority daemon on the Kubernetes master. That authority creates a cluster CA certificate and signs subordinate certificates for the components.

The certmgr daemon then monitors active certificates and renews them when needed. This is more than generating a static configuration file once. The declared system includes processes that maintain the validity of the credentials required by the cluster's ongoing communication.

For the single-node case, enabling easyCerts is sufficient for the certificate setup described by the chapter. Joining additional nodes requires establishing initial trust. The module cannot simply assume that any machine asking for a certificate belongs to the cluster.

The supplied nixos-kubernetes-node-join helper takes a token on standard input, copies it to the Kubernetes secrets directory and restarts certmgr. As the requested certificates become available, it restarts components so they pick up the new key pairs. This sequence identifies the initial enrollment operation that remains necessary when extending the cluster.

The chapter also states that easyCerts does not support multi-master high-availability clusters. That is a scope limit on this automatic arrangement. It should not be presented as a complete PKI solution for every possible Kubernetes topology merely because it handles the documented single-master case.

I find this account useful because it makes the operational requirement visible. Automatic renewal is part of the service configuration, while initial trust is a separate enrollment step. A declaration can coordinate those mechanisms without making the trust decision disappear.

Insecure ports stop being the default

The Kubernetes API server and controller-manager port options are renamed to insecurePort and default to zero, disabling those endpoints. The release chapter explains that the ordinary plain HTTP component ports are disabled by default. An administrator can still configure them explicitly.

At the same time, the API server's bindAddress default changes from loopback to 0.0.0.0 so it can be reached from outside the master itself. Those two changes need to be read together. External reachability is part of a distributed cluster, while the default listener should use the authenticated communication model.

The notes recommend binding an explicitly enabled insecure API endpoint only to loopback. That recommendation concerns a separate insecureBindAddress setting. Confusing the secure listener's broader binding with permission to expose an insecure listener would invert the purpose of the refactoring.

Privileged containers are also disallowed by default through the API server and kubelet allowPrivileged settings. That is another policy decision, separate from whether a client is authenticated. The release makes several concrete default choices instead of treating the presence of TLS as the complete security configuration.

Administrative credentials remain privileged material

When easyCerts is enabled, the module generates a cluster-admin kubeconfig and links it at /etc/kubernetes/cluster-admin.kubeconfig. The chapter says the referenced key pair belongs to root, so the default file gives administrative access through root on the master.

That is a local privilege boundary as well as a convenience for administration. The generated file is not an ordinary example configuration that every cluster user should share. Its referenced credentials establish the broad permissions that the manual associates with cluster administration.

The arrangement also shows why generating configuration and distributing authority are different tasks. NixOS can produce the file and make its location predictable. Deciding who receives access to the underlying key pair remains a decision about cluster privileges.

The release extends the same explicitness elsewhere

Koi adds systemd service confinement options that can restrict a service to a chroot containing store paths from its runtime closure. This addresses filesystem visibility for a running service, while the Kubernetes refactoring addresses authenticated component communication. They operate at different boundaries.

The release also switches the Kubernetes DNS addon to CoreDNS and defaults it to two pod replicas. Existing clusters may see a brief disruption while the addon terminates and starts again. The configurable replica count is services.kubernetes.addons.dns.replicas. That change has a lifecycle effect even though the resulting desired configuration is declarative.

Pantheon is another visible addition, enabled through services.xserver.desktopManager.pantheon.enable. It illustrates the range of services the module system configures, but the Kubernetes work is the more substantial example of coordinating identities across processes. For a newly configured cluster, the default now includes the certificate and authorization relationships its components require, with an explicit master address and a documented enrollment path for additional nodes.