Kueue admits batch work before Kubernetes schedules its pods

Creating every batch pod immediately is an expensive way to express that most of the work must wait. Kubernetes' Job controller normally creates pods after a Job appears, and the scheduler attempts to place them. At scale, a backlog becomes work for the control plane before anyone has decided which tenant should receive the available capacity.

The October 4 introduction of Kueue addresses that earlier decision. Kueue manages admission of a batch job as a unit. It decides when the job may begin and which resource flavor it may use, while leaving pod placement and Job lifecycle management to existing Kubernetes components.

I prefer this division because it extends Kubernetes through controllers without replacing working control loops. Queue admission and pod scheduling answer different questions. Keeping them separate lets a batch policy operate at the job level without making it responsible for every node assignment.

ResourceQuota does not provide a waiting room

The introduction distinguishes batch queueing from ResourceQuota. ResourceQuota enforces limits when resources are created. It does not queue rejected creation requests and later admit them in a selected order. That makes it useful for protecting a cluster but insufficient for the batch-sharing model described here.

A batch system needs to represent work that has been submitted but has not yet received permission to consume its full allocation. It also needs to express how tenants share unused quota and how a job can use different kinds of capacity. Repeatedly attempting pod creation does not express those policies directly.

The distinction remains relevant in an autoscaled cluster. The introduction notes that cloud capacity still has quota and cost constraints. Autoscaling changes how physical capacity appears, but it does not decide an organization's intended allocation of that capacity among tenants.

Kueue's placement in the control flow follows from this problem. The job remains the unit awaiting admission. Pods become scheduling work after the queueing decision permits the Job to run. That keeps the backlog represented at the level where the sharing decision is made.

Job suspension supplies the lifecycle control

The original suspended Jobs introduction explains the spec.suspend field. A suspended Job does not cause its controller to create new pods. Setting the field back to false lets the controller proceed.

That account also explains why deleting a Job is an inadequate substitute for suspension. Deletion loses the Job's completion history and associated information. Suspension keeps the object and its existing record, with lifecycle semantics designed for stopping and resuming work.

Kueue uses this capability to govern admission. The October introduction describes intercepting a submitted Job and suspending it. When quota permits admission, Kueue releases the Job through the same field. The Job controller remains responsible for creating and tracking the resulting pods.

The second required capability is mutable scheduling directives before the Job starts. Kueue can set the pod template's node selector to match the selected resource flavor. This lets it influence the resource type without choosing a particular node.

These controls define an integration contract for custom batch APIs too. The introduction says another Job API can participate if it offers suspension and mutable placement directives. Integration therefore depends on expressible lifecycle controls, not on replacing that API with Kueue's own workload implementation.

Flavors describe available kinds of capacity

The resource model includes ResourceFlavor, ClusterQueue, and LocalQueue. ResourceFlavor describes a type of capacity using labels corresponding to nodes. The introduction's example distinguishes spot and on-demand provisioning through node labels, with spot nodes also carrying a taint.

ClusterQueue defines resource pools through quotas associated with flavors. The example gives on-demand CPU capacity an earlier position than spot capacity. Flavor order matters because Kueue attempts to fit jobs according to that order unless the workload explicitly constrains its choices.

LocalQueue supplies the namespaced entry point for a tenant. In its simplest form, it points to the ClusterQueue that the namespace can use. Users discover an available queue without needing to manage the cluster-scoped pool themselves.

The distinction separates administration from submission. An administrator defines the resource pools and their limits. A user submits a Job through a named local queue. The objects connect those responsibilities while retaining namespace boundaries around the user's work.

A LocalQueue from the introduced API can be expressed as follows. This uses the v1alpha2 schema shown in the announcement, not a later Kueue API.

apiVersion: kueue.x-k8s.io/v1alpha2
kind: LocalQueue
metadata:
  name: training
  namespace: team-ml
spec:
  clusterQueue: research-pool

A submitted Job identifies this queue using the kueue.x-k8s.io/queue-name annotation in the announcement's example. Its requested resources and tolerance for spot capacity affect which flavor can admit it. The queue name alone does not make incompatible placement requirements disappear.

A quota admission decision precedes actual node provisioning and pod placement

In the example, Kueue checks whether the Job's requested resources fit available quota. If on-demand quota is consumed but spot quota remains, a Job that tolerates spot capacity can receive that flavor. Kueue then updates suspension and the node selector in one Job update.

The Job controller creates the pods. Kube-scheduler attempts placement on matching nodes. If sufficient matching capacity does not exist, the scheduler marks the pods unschedulable and the cluster autoscaler can provision nodes.

This sequence separates allocated quota from existing free machines. A job can be admitted under a resource policy while waiting for the physical capacity that satisfies its placement constraints. Kueue's admission does not mean that its pods are already running.

My interest is in that explicit sequence of responsibilities. The queue controller chooses when a job may compete for capacity and which flavor it may consume. Existing scheduling and autoscaling components handle the remaining work through their established interfaces.

As of the introduction, Kueue is an early project requiring Kubernetes 1.22 or newer. The authors identify preemption as immediate future work, with hierarchical quota, budgets, and dynamically sized jobs also planned. Those are prospective capabilities. The present design supplies job-level admission through suspension, quota, and flavor selection while retaining the Kubernetes components that execute the admitted work.