August 4, 2021·6 min read·engineeringkubernetesapicontrollers

Kubernetes 1.22 makes field ownership stable and removes beta APIs

Server-side apply reaches general availability in Kubernetes 1.22. Its ownership rules help multiple writers share objects, while beta API removals require clients and manifests to use stable endpoints.

A Kubernetes object often has several writers. A deployment tool declares application configuration while controllers update the parts they manage. Server-side apply makes those claims explicit at the field level, and Kubernetes 1.22 promotes that mechanism to general availability.

The 4 August release announcement also removes several deprecated beta APIs. These changes address different parts of the API contract. Server-side apply defines how clients collaborate on an object's contents. API removals define which versioned endpoints those clients may use to reach it.

I consider the API machinery more consequential than any single built-in workload. Field ownership matters because controllers and declarative deployment systems routinely need to contribute to the same resource. Making that collaboration visible in the API gives their disagreements a concrete representation.

An apply request states a manager's intent

The server-side apply documentation available before this release describes a fully specified intent as a partial object. A manager includes the fields whose values it cares about. The server combines that intent with the live object and records field management information.

Fully specified does not mean copying every field returned by a read. It means declaring the complete set that this workflow intends to manage. Fields supplied by unrelated controllers should not become part of a deployment workflow's intent merely because they appear in the live object.

The API records ownership in metadata.managedFields. A manager identifies a workflow, and apply requests must provide a field-manager identity. That identity lets the server distinguish two workflows even when both operate on the same resource.

Apply uses the PATCH endpoint with the application/apply-patch+yaml content type. The documentation says to use that content type even when the submitted document is JSON, since JSON documents are valid YAML. The operation's semantics come from the apply mechanism, not from treating it as an ordinary update.

This matters for controller implementation. A controller can apply the fields it manages without first reading the entire object and constructing a complete replacement. The documentation still requires the applied object to contain all the fields the controller cares about. Partial intent must be complete with respect to that manager's responsibility.

Conflicts identify competing claims

When an apply request tries to change a field owned by another manager, the server reports a conflict. The conflict warns that accepting this intent would undo another workflow's declared value. It supplies information that a simple last-writer outcome would not express.

A manager can resolve the disagreement by giving up that field. Removing it from the applied configuration relinquishes the management claim. Alternatively, the manager can match the existing value and share ownership with the other manager.

Forcing the apply has a different effect. The server accepts the requested value and removes the conflicting ownership claims from other managers. That is an explicit transfer of responsibility, so it changes the collaboration contract as well as the field's contents.

Shared ownership also has defined behavior. Two managers applying the same value can both own the field. A subsequent attempt to change that shared value conflicts. This lets a workflow hand responsibility to another without an intervening change in value.

Ordinary update operations behave differently. The documentation says both apply and update can alter managed fields, but only apply rejects these ownership conflicts. Field ownership is therefore not a general authorization rule that prevents every permitted client from changing an object.

Omission is an operation with consequences

Removing a field from an applied configuration can delete it from the live object or restore its default when no other manager owns it. This gives omission meaning. A manager cannot assume that dropping a field merely stops mentioning it while preserving its value.

The documentation's handover example concerns Deployment replicas and the HorizontalPodAutoscaler. A user wants the autoscaler to own the replica count. If the user removes that field before the autoscaler has written it, the count can revert to its default.

The documented solutions coordinate ownership. One approach waits until the autoscaler has claimed the field. Another uses a temporary manager to share ownership before the original configuration drops it. The shared claim preserves the value during the handover.

This is a useful improvement in expressiveness, but it requires understanding the rules. Declarative configuration describes continuing responsibility, so removing a field can be as consequential as changing its value. A repository diff that deletes replicas can therefore represent a transfer protocol involving another controller.

For custom resources, schema design affects this collaboration. The documentation distinguishes atomic collections from collections whose entries can be managed independently. A list represented as atomic belongs to a manager as a whole, while suitable map or set annotations permit separate entry ownership.

API endpoint removal does not erase every object

The 14 July migration explanation lists the beta API versions that 1.22 stops serving. They include beta Ingress, CustomResourceDefinition, admission webhook configuration, and several authentication and authorization APIs.

Existing objects remain accessible through their stable APIs. For example, an Ingress created using an older API version can be retrieved or updated through networking.k8s.io/v1 on a version that serves it. The API version used by a manifest is not the object's only possible representation.

The distinction is between retaining access to resource data and continuing to serve every historical request format. The release preserves the former through stable APIs while retiring the deprecated beta endpoints.

That does not make every client compatible automatically. A controller that continues requesting a removed beta endpoint still needs a code change. A deployment tool that repeatedly submits beta-version manifests also needs an update even when the corresponding objects already exist.

This is why both live configuration and client behavior matter. Converting stored manifests cannot repair a binary that constructs old API requests internally. Updating a binary cannot repair an old manifest that a separate workflow continues to submit.

The migration explanation also warns about functional differences. A mechanical conversion can produce a structurally updated object while leaving decisions that deserve explicit review. Ingress path handling is one example because the stable API makes path types visible.

Stable Ingress makes matching intent explicit

The migration post recommends the stable Ingress API, available since 1.19. It also recommends moving from the legacy ingress-class annotation to spec.ingressClassName where appropriate.

Older Ingress definitions use ImplementationSpecific path behavior. The post recommends considering Prefix or Exact when converting rules. Those choices express different matching intent and can make movement between Ingress classes easier.

An ingress controller must also support the stable API. A manifest alone does not implement incoming request handling, so updating its version is only part of the compatibility change. The controller consuming it must understand that representation.

The same general issue applies to CRDs and webhooks. They are part of the API's extensibility machinery, and consumers need to use the stable resource definitions and supported request formats. API cleanup therefore reaches beyond application manifests into the components extending the cluster.

Stability applies to specific mechanisms

Kubernetes 1.22 also promotes external client credential providers to stable. The release announcement identifies better support for interactive login flows and bug fixes as part of that graduation. This concerns client authentication integration, distinct from how apply manages object fields.

Other capabilities remain alpha. The announcement introduces PodSecurity admission as an alpha replacement effort for PSP and describes memory QoS using cgroups v2 as alpha. A release that stabilizes one mechanism can simultaneously introduce experiments elsewhere.

The ephemeral-container API is another compatibility boundary. Its API changes in 1.22, and the announcement says kubectl 1.21 or earlier cannot create ephemeral containers through the new mechanism. The usual compatibility expectations for stable capabilities do not make an alpha API change disappear.

My view is that 1.22 strengthens the declarative model by exposing field responsibility while requiring clients to complete migrations to established stable APIs. These are concrete contracts that controller authors can implement. An apply manager must declare the fields it owns, and an API client must request a version the server still serves.

For existing automation, the resulting work is specific. Ownership conflicts need deliberate resolution, omitted fields need understood semantics, and dependencies that still call removed beta endpoints need replacement requests. Kubernetes retains the objects through their stable representations, while making the responsibilities of their writers more explicit.