Kubernetes 1.18 coordinates CPU and device locality

Allocating enough CPUs and a suitable network device does not guarantee that a latency-sensitive workload receives a good allocation. On a machine with multiple NUMA nodes, the CPUs and device can belong to different parts of the system. Independent allocation decisions can satisfy resource counts while introducing the locality cost the application was trying to avoid. Kubernetes 1.18 promotes Topology Manager to beta to coordinate those decisions inside the kubelet.

The March 25 release announcement identifies CPU and device alignment as a major change. It also introduces kubectl alpha debug, a command for attaching an ephemeral debugging container to a running pod. Both changes address cases where an abstract workload description needs a more precise relationship with the machine executing it. They operate at different stages, allocation before execution and inspection after a problem appears.

I value Kubernetes for its ability to express operational state through APIs and controllers. Hardware locality tests the limits of that abstraction. A pod's resource count cannot express every property of the eventual allocation, and the node needs a way to reject an allocation that cannot meet the chosen policy. Topology Manager makes that decision explicit.

Resource managers exchange topology hints

Before Topology Manager, the CPU Manager and Device Manager made their allocation decisions independently. The topology documentation linked by the release announcement describes an interface through which these components become hint providers. Each supplies topology information as a NUMA node bitmask and an indication of whether an allocation is preferred.

Topology Manager combines those hints under a node policy and records the selected affinity. The resource managers use that information when they make their assignments. This is coordination between allocators inside the kubelet. It does not make the cluster scheduler aware of the detailed topology on every candidate machine.

Consider a container requesting dedicated CPUs and a device advertised through an extended resource. The CPU allocation can be locally acceptable in isolation, and the device allocation can also be locally acceptable. Their combination may still require access across NUMA nodes. Exchanging hints gives the kubelet a basis for selecting an aligned combination or refusing admission under a stricter policy.

That distinction matters when interpreting an allocation failure. A node can have sufficient aggregate resources while lacking a combination acceptable to its topology policy. Resource availability is therefore constrained by the relationship between the available resources. Increasing a count without changing that relationship may not make the same workload admissible.

Beta does not select a locality policy

The feature gate is enabled by default in 1.18, but the default topology policy is none. Under that policy, Topology Manager performs no alignment. An upgrade therefore does not silently impose strict NUMA placement on ordinary workloads. Operators who want alignment must select a policy and meet the relevant resource-manager requirements.

The documented policies are none, best-effort, restricted, and single-numa-node. Best effort records the preferred affinity but admits the pod even when the selected affinity is not preferred. Restricted admission rejects the pod if the selected hint is not preferred. Single NUMA node admission requires that a single-node affinity be possible.

The relevant kubelet option can be expressed as follows. This is one policy setting, not a complete kubelet configuration or a complete recipe for dedicated CPU allocation.

--topology-manager-policy=single-numa-node

The documentation describes using the static CPU Manager policy and workloads making CPU or extended-device requests. For a Guaranteed pod requesting CPUs and a device, CPU Manager can contribute CPU topology hints while Device Manager contributes the requested device's hints. A device-only workload can still contribute useful device hints without a CPU request. The selected policy determines how the combined information affects admission.

Best effort and strict admission make different availability choices. Best effort permits execution when the preferred alignment cannot be achieved. A strict policy can preserve an application's placement requirement by declining to run it on that node. Neither choice can create hardware combinations that the machine does not have. The appropriate policy depends on whether the application tolerates a less favorable allocation.

Rejection happens after scheduling

The topology documentation states that the scheduler is not topology aware. It can assign a pod to a node that subsequently rejects it through Topology Manager. That ordering is a material limitation of the feature. Cluster placement and local resource admission do not share the same information in this release.

The documented rejection leaves the pod terminated with an admission failure. The scheduler does not then reschedule that same terminated pod. A ReplicaSet or Deployment can create a replacement, and an external control loop can respond to topology-affinity errors. Workload management therefore affects what happens after a locally unacceptable placement.

This is a specific place where I care about the distinction between reconciliation and a single scheduling attempt. A controller can create another pod after failure, but repeated creation does not establish that any node can satisfy the resource combination. Topology admission exposes a constraint that the workload owner still has to accommodate in the available hardware and placement policy.

There are further boundaries in the documented beta. CPU Manager and Device Manager implement the hint-provider interface. Memory and hugepage allocations do not participate in NUMA alignment through Topology Manager. The documentation also limits supported machines to eight NUMA nodes because enumerating affinities beyond that causes a state explosion. The feature's name should not be read as a claim that every topology-sensitive resource is coordinated.

For example, admitting a container with CPU and device alignment does not establish a corresponding guarantee about the locality of its memory. That follows from the documented set of participating hint providers. An application whose latency depends on all of those allocations still has requirements outside this beta's coordination. Reading the policy alongside that resource coverage gives a more accurate account of what admission success means. It establishes that the participating managers found an acceptable combination under the selected policy.

Debugging can add tools to the running pod

Kubernetes 1.18 also adds the alpha kubectl alpha debug command. The release-time changelog describes attaching an ephemeral container to a running pod for debugging. The release announcement explains that the command can attach an interactive console to the temporary container.

The distinction from an ordinary command execution is useful. Executing a command in a container presupposes that the container has a suitable command available. An ephemeral debugging container provides a separate place for diagnostic tools while the target pod remains the subject of investigation. The release presents this as alpha functionality, so the command should be understood in that maturity context.

This is a separate response to the gap between a workload declaration and runtime behavior. Topology policy influences whether a particular allocation is acceptable before a container starts. A debugging container helps inspect a pod after it exists. The release gives operators more control at each point without claiming that the two mechanisms diagnose or solve the same class of problem.

Change inspection gets more precise

The same release promotes kubectl diff and API server dry run to stable. Its changelog records a useful distinction in the diff command's result codes. A result of one identifies a detected difference, while a result greater than one identifies a kubectl error. Automation can distinguish changed desired state from a failed attempt to inspect that state.

The dry-run flag now accepts client, server, and none. The changelog deprecates the former boolean and unset values. That gives the invocation an explicit choice about where the dry-run strategy operates. It is a small command-line change with consequences for scripts whose argument construction previously treated dry run as a simple switch.

The announcement also describes a second beta for server-side apply, which tracks and manages field changes for new objects. Ingress gains pathType and IngressClass, with ingressClassName replacing the deprecated class annotation. These changes make previously implicit choices more visible in API objects and command behavior. They remain separate from any claim that all controllers implement identical semantics.

For hardware-sensitive workloads, the immediate decision in 1.18 remains the node's admission policy. Enabling the feature gate leaves alignment off under the default policy. Selecting a stricter policy moves a locality requirement into kubelet admission, where an unsatisfied requirement can terminate the pod after scheduling. Workload controllers and hardware capacity planning must account for that documented result.