September 18, 2019·6 min read·engineeringkubernetescrdsapireleases

Kubernetes 1.16 makes API contracts enforceable for extensions

CRDs reach general availability with stronger schema requirements in Kubernetes 1.16. The release also stops serving older workload APIs, demonstrating the version lifecycle that extension authors now need to design for.

Kubernetes 1.16 makes two related statements about its API. Extensions can define stable custom resources through apiextensions.k8s.io/v1, and clients can no longer assume that deprecated workload endpoints will remain available indefinitely. The September 18 release announcement describes general availability for CustomResourceDefinitions alongside rejection of requests using removed API versions. Together they clarify the contract between an API author and its clients.

The first change gives extension authors stronger machinery for maintaining that contract. The second demonstrates why the contract needs a lifecycle. A resource can keep its meaning and persisted data while the version through which a client accesses it changes. Designing a custom resource now includes choosing its schema and version-conversion behavior with those relationships in mind.

I regard custom resources and controllers as central to Kubernetes' usefulness. Their general availability matters because an extension should be understandable through the same API expectations as a built-in type. It also puts more responsibility on extension authors to define their data precisely instead of relying on the server to preserve arbitrary objects indefinitely.

Stable custom resources require a stronger description

The release announcement traces CRDs to a redesign of ThirdPartyResources and says they have been beta since Kubernetes 1.7. General availability adds requirements and default behavior intended to protect consistency for clients. Structural schemas and pruning make the declared shape of an object consequential to what the server stores. Defaulting is enabled by default for the new CRD API.

The version-pinned v1 CRD definitions show how a resource is described. Its group and plural name establish its API identity. Scope selects a cluster-scoped or namespaced resource. A versions list defines the representations that the server can expose, with separate flags for serving and storage.

Each version can describe the schema used for validation, pruning and defaulting. This joins several server operations to a common definition. A schema supplies the recognized data, validation restricts acceptable input, and defaulting fills unspecified values according to declared rules. Pruning prevents unknown data from quietly becoming part of the stored interface.

That arrangement changes the role of schema maintenance. An omitted field is no longer merely absent from generated documentation. With pruning, it may be absent from the object returned after persistence. A default also becomes part of the interface because clients that omit a field can receive a server-supplied value. Extension authors need to account for both behaviors when evolving an existing resource.

The release describes protection of the *.k8s.io group as another guardrail. Its broader point is that the stable API makes several previously optional protections harder to miss. General availability concerns a defined extension mechanism with consistency rules, rather than a promise that every custom API written on top of it is automatically well designed.

Served versions and storage versions have different jobs

The v1 types require exactly one version marked for storage. Other entries can be served without being the representation selected for persistence. This distinction allows clients to use different supported versions while the server maintains a defined stored form. The custom resource's version list is therefore an evolution mechanism, not a set of independent databases. Discovery ordering also uses version names, preferring Kubernetes-style version names according to the ordering rules documented in these types.

Conversion specifies how those representations relate. The None strategy changes the API version without transforming other fields. A webhook strategy calls an external conversion implementation. The latter requires the webhook configuration and unknown-field preservation to be disabled. Choosing the strategy depends on whether the representations require real data transformation.

The webhook configuration also lists preferred ConversionReview versions, and the API server uses the first supported one. If no version is supported, conversion fails. A custom resource therefore has its own client-facing versions while its conversion service has a separate protocol compatibility requirement. Both matter to the ability to serve the resource successfully.

This fragment illustrates the distinction between serving and storage in an otherwise complete CRD. It is deliberately limited to the version flags, so schemas and the rest of the definition must be supplied separately.

spec:
  versions:
    - name: v1beta1
      served: true
      storage: false
    - name: v1
      served: true
      storage: true

The names in this illustration are choices for an extension, not a claim that Kubernetes supplies these versions for every custom type. Both representations are available to clients, while only one is the selected storage form. Whether conversion can simply relabel the object or must transform its data depends on the schemas and behavior the extension author defines.

The webhook becomes part of ordinary API operation

An external conversion implementation needs an address and certificate trust. The API definitions allow a service reference or an HTTPS URL, with a CA bundle to validate the webhook server certificate. An in-cluster webhook should use the service reference. A loopback address carries a warning because the webhook must then run on every API server host that may call it.

These details make conversion an operational component of the API. Declaring two versions is insufficient when requests between them need a service that cannot be reached or whose certificate cannot be validated. The conversion server's deployment and connection configuration support behavior that API clients can encounter during routine reads and writes.

I prefer that these dependencies are explicit in the resource definition. They make it possible to identify which implementation owns the conversion and how the API server reaches it. The operator still has to maintain that implementation, just as the API author has to preserve the meaning of fields across versions.

Removed workload endpoints demonstrate the other side of stability

The release announcement gives a concrete example of the API lifecycle. A Deployment submitted as extensions/v1beta1 now receives a no-matching-kind error. Kubernetes has stopped serving that deprecated endpoint. The July migration notice identifies apps/v1 as the replacement for the affected workload resources and states that existing persisted data remains accessible through replacement versions.

That means removal affects clients and manifests even while the workload object remains present. A deployment tool using an old endpoint needs to change its requests. A controller compiled against an obsolete API can retain the same dependency without an old YAML file being visible in the operator's repository. The migration concerns all callers of the removed versions.

Changing the version can also expose field differences. July's notice identifies required immutable selectors and changed rollout defaults for workloads. The stable replacement's semantics apply when a client submits the new representation. This is why a migration cannot be evaluated solely by whether a version string has been replaced successfully.

The comparison with CRD general availability is practical. Extension authors now have the mechanisms to serve multiple versions and convert between them. Kubernetes' own API transition shows why clients need time to move and why old representations cannot be treated as an indefinite obligation. A stable extension needs a version policy as well as a schema that parses.

EndpointSlices address the cost of watching large objects

The release also introduces an alpha EndpointSlice API. Its motivation is another consequence of API design. A conventional Endpoints object contains all endpoints for a Service. When one endpoint changes, every watcher receives the full updated resource. Kube-proxy running on each node multiplies the amount of data transmitted as services and clusters grow.

EndpointSlices divide the endpoints into smaller objects, limited by default to 100 endpoints each according to the announcement. Updating one slice then transfers a subset of the Service's endpoint data to watchers. The mechanism targets update amplification caused by the size of the watched object. It is relevant even when the logical change is only one pod becoming available or disappearing.

This API remains alpha and disabled by default in 1.16, with alpha kube-proxy support. The existing Endpoints API continues as the default. That maturity distinction separates an experiment in scaling observation from the generally available CRD mechanism. Both changes show that an API's structure affects operational behavior, including what the server must transmit and what clients can depend on.

For a cluster adopting 1.16, the immediate API boundary is concrete. Old workload endpoints reject requests, while v1 CRDs offer stronger rules for defining extensions. A custom API's supported representations, structural schema and conversion service now provide the means to maintain consistent behavior over time. Clients still need to use the versions that their server actually serves.