June 14, 2023·4 min read·engineeringkubernetescontrollersapi-machinery

Kubernetes 1.27.3 fixes API registration during startup

A startup fix in Kubernetes 1.27.3 prevents premature APIService deletion for custom resources. The underlying problem shows why an empty controller cache cannot always be treated as an authoritative absence.

A controller starting with incomplete information can make a destructive reconciliation decision before it learns what exists. Kubernetes 1.27.3 includes a fix for that problem in API registration. At API server startup, APIService objects associated with custom resources could be deleted and recreated. The intended protection already involved waiting for an initial custom resource definition sync, but the code checked whether that wait was needed through the wrong server configuration.

The June 14 maintenance release carries the correction. Its startup synchronisation patch makes the CRD-enabled decision in the API extensions configuration and passes the result to the aggregation server. That is a small code change with a specific ordering requirement. Automatic registration must not interpret an incomplete initial view as evidence that existing registrations should disappear.

I regard the API machinery as Kubernetes' most useful contribution. Watches and custom resource definitions make it possible to build controllers around declared state. That design also makes startup ordering important, because a controller needs to distinguish an authoritative empty result from information it has not yet loaded.

The wait existed but its condition was wrong

The patch's existing comment explains the intended sequence. The CRD registration controller should process the initial set of custom resource definitions before the automatic registration controller begins its initial sync. This prevents that initial sync from deleting APIServices for CRDs that still exist. Waiting is necessary only when the server enables the CRD API.

Before the fix, the condition examined aggregatorConfig.GenericConfig.MergedResourceConfig. It asked that configuration whether the custom resource definition resource was enabled. The patch moves the resource-enabled check into the API extensions server configuration, which is the configuration relevant to that API. It passes a crdAPIEnabled boolean into createAggregatorServer.

This changes the source of the decision without changing the purpose of the wait. The aggregator still calls crdRegistrationController.WaitForInitialSync() when CRDs are enabled. Automatic registration starts after that call returns. When the CRD API is disabled, registration can proceed without waiting for a sync that is unnecessary for that configuration.

The distinction is about ownership of configuration. A server participating in an API chain does not necessarily provide the authoritative answer about every API in that chain. Reading the correct configuration matters even when the resource name and the predicate look reasonable in isolation. The patched code makes that dependency an explicit argument.

Initial absence needs a different interpretation

The faulty sequence creates a gap between persistent state and a controller's initial knowledge of it. If automatic registration acts before the CRD registration controller has processed existing definitions, the information needed to preserve their registrations is incomplete. The patch comment names the resulting deletion risk directly.

This is a useful example of a reconciliation precondition. Eventual convergence does not mean that every intermediate action is desirable. Deleting and later recreating an object can still be an incorrect consequence of observing too early. The correct response at this stage is to wait for the initial information needed to make the decision.

The fix therefore preserves an ordering relationship between two controllers. It does not rely on later events to reverse an avoidable deletion. Once the initial CRD sync completes, automatic registration can reconcile using the information that the startup sequence was supposed to provide from the beginning.

That interpretation follows the patch itself. It does not require assuming that custom resource data was deleted, or that every API server restart produced a visible application outage. The stated bug concerns deletion and recreation of APIService objects. Keeping the claim at that level gives operators a concrete behaviour to associate with this patch without expanding it into an unsupported failure narrative.

The patch makes the startup boundary observable

The implementation adds log messages before and after the initial CRD sync wait. It also logs the case where the CRD API is disabled and APIService registration starts without the wait. Those messages expose which branch the startup sequence took, instead of leaving the ordering decision entirely implicit.

The same pull request includes integration test changes for safe APIService handling during startup. The code adds a test-server option to skip its ordinary health wait, which allows tests to exercise startup states that cannot be examined after waiting for normal health completion. That fits the bug's timing. A test that begins only after startup has settled can miss an incorrect action during initialisation.

I find this kind of maintenance change particularly relevant to controller design. A missed event and an incomplete initial view are different conditions. Reconciliation can recover from a missed event once it has a reliable current view, but it should not make deletion decisions while still assembling that view.

Kubernetes 1.27.3 supplies the corrected configuration check and preserves the intended wait. The important dependency is now explicit in the call between server components. Automatic API registration begins after initial CRD processing when the CRD API is enabled, and it can proceed directly when that API is disabled.