Kubernetes 1.6 makes shared clusters more practical

A cluster shared by several applications needs more than enough capacity to run them. Its users need permissions that do not grant control over their neighbors. Workloads need placement rules, and storage requests need a path to actual volumes. The Kubernetes 1.6 announcement brings those concerns together under support for multiple users and workloads at scale. The increase to 5,000 supported nodes is the most visible number, but the changes around that number explain what makes a larger cluster usable.

Dynamic storage provisioning and StorageClass become stable. Role-based access control reaches beta, along with kubeadm and several scheduling mechanisms. These changes reduce different kinds of coordination between application owners and cluster administrators. The release also supplies defaults that perform more work automatically, which means a request that once required manual preparation can now cause infrastructure to be created and later deleted.

I value Kubernetes for the relationship between declared state and the controllers that act on it. This release makes that relationship more useful for application owners. It also makes the administrator's choices about defaults more consequential, because those choices determine what a short application declaration will actually do.

A storage request can allocate infrastructure

PersistentVolume and PersistentVolumeClaim separate a workload's request for storage from the specific storage implementation. That separation permits a Pod specification to refer to a claim without embedding the details of the underlying provider. Before considering dynamic provisioning, the distinction already gives administrators a place to manage storage independently of application manifests.

Dynamic provisioning extends the model to the creation of the backing storage. An administrator defines a StorageClass with a provisioner and provider-specific parameters. An application requests storage through a claim, and the provisioning machinery can create a volume that satisfies it. The release announcement describes multiple classes as a way to expose different storage options while keeping their implementation details with the administrator.

The claim definition in the 1.6 source includes storageClassName, access modes and a resource request. A claim for a class that the administrator has already configured can be concise. The following example assumes that standard exists in the cluster and has a functioning provisioner.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: application-data
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: standard
  resources:
    requests:
      storage: 10Gi

The class name is an administrative choice, not a portable promise that every cluster offers the same storage. ReadWriteOnce requests read-write mounting on one host, as defined in the version's API types. The declaration says what the application needs while leaving the provisioner's actual volume operation outside the Pod specification. Its usefulness depends on the class having the intended meaning in the target environment.

Kubernetes 1.6 preinstalls system-defined StorageClass objects by default for AWS, Azure, GCP, OpenStack and VMware vSphere. The announcement explicitly calls this a change in default PVC behavior on those providers. Application owners can benefit from dynamic provisioning without first arranging for an administrator to create a class. Operators who previously relied on manual provisioning need to account for the new default when interpreting a claim.

Deletion has a similarly concrete meaning. The announcement says dynamically provisioned volumes use the Delete reclaim policy by default. Deleting the claim therefore also deletes the dynamically provisioned volume. That is convenient for disposable storage and materially different from leaving a disk for someone to reclaim later. The storage interface includes the end of the volume's life as well as its creation.

Permissions accompany the automation

RBAC reaches beta with more tightly scoped default roles for system components. The default policies grant permissions to control-plane components, nodes and controllers. Administrators can grant users and service accounts fine-grained access to particular resources within a namespace. That is relevant to a shared cluster because the authority to submit one application's resources should not imply authority over every application's resources.

The division of responsibility becomes clearer when storage and authorization are considered together. A claim can ask a controller to allocate infrastructure, but the right to create claims is still an API permission. Automation removes a manual action without removing the need to decide who can initiate it. RBAC provides a way to express that decision in terms of the resources that the application owner actually uses.

Kubeadm also reaches beta, with RBAC setup included in its base functionality. The release lists the Bootstrap Token system and an improved Certificates API among its bootstrap capabilities. An installation tool consequently participates in the initial security arrangement, instead of leaving authorization entirely as work to perform after the cluster starts.

This develops the direction set out in the January cluster lifecycle roadmap. That roadmap called for enabling RBAC and reducing the permissions of kubelet and built-in services. The March announcement supplies a specific release status for the resulting setup and authorization work. It does not imply that every roadmap proposal has shipped or that kubeadm now provisions machines.

Placement policy has several independent meanings

The scheduling changes address requirements that a replica count cannot express. Node affinity and anti-affinity reach beta and let a workload use node labels to constrain placement. Required rules impose a condition on scheduling, while preferred rules describe a preference. The distinction lets an application express whether locality is necessary for operation or merely desirable.

Taints and tolerations, also beta, describe exclusions from the node side. They can reserve a group of nodes for particular users or keep specialized hardware available for applications that need it. This is a different administrative concern from an application asking for a node label. A shared cluster needs a way for the owner of the nodes to restrict use even when other applications might otherwise fit there.

Pod affinity and anti-affinity express placement relative to other Pods. The release describes both co-location for communication and separation for failure tolerance. Those rules can operate over topologies such as nodes or zones, with hard or soft requirements. A workload's desired placement can therefore depend on what else is running, not only on a static description of the machines.

Multiple schedulers reach beta as well. Each scheduler handles its own set of Pods, allowing a custom scheduling policy to coexist with the default scheduler. That option is useful when the built-in placement language does not capture an application's requirements. It also gives the cluster another component whose assignment of work must remain explicit.

These mechanisms should not be collapsed into a single promise that the scheduler will understand an application's architecture. They describe particular placement constraints and preferences. Application owners still have to select the topology and strength that correspond to the behavior they need. A required condition that the available nodes cannot satisfy leaves less freedom than a preference.

The larger cluster depends on shared machinery

The supported scale increases to 5,000 nodes and 150,000 Pods under the project's scalability service-level objectives. The announcement attributes the increase in part to etcd v3. New clusters use the etcd v3 storage backend by default, while the release advises administrators upgrading a 1.5 cluster to plan a data migration window. The default for a new installation and the work required for an existing installation are different matters.

The release also makes the Docker implementation of the Container Runtime Interface beta and enables it by default in kubelet. Alternative runtime integrations remain alpha in the announcement. These statuses matter because a common runtime interface does not mean every implementation has the same maturity. Kubernetes can standardize the kubelet boundary while individual runtime integrations continue to develop.

Federation addresses a separate scale requirement. The beta kubefed utility helps combine clusters behind a single API endpoint and improves support for on-premise installations. That provides an option for multiple regions or clouds without treating every such deployment as one larger cluster.

Node Allocatable lets an administrator reserve compute resources for system daemons. A node's total capacity cannot all be promised to applications if the operating system and Kubernetes services also need to run. That reservation connects the control plane's placement decisions with the resources the node must retain for its own operation.

My interest in the 5,000-node limit is therefore conditional on these surrounding changes. Bigger clusters put more applications under the same administrative decisions. Kubernetes 1.6 supplies more ways to scope authority, express placement and provision storage within that shared system. The default deletion behavior for dynamically provisioned volumes is a particularly clear example of why those administrative decisions belong in the platform's application-facing contract.