May 20, 2024·6 min read·engineeringkubernetesarchitecturecloud

Kubernetes cloud migration required four replacement systems

Removing built-in cloud integrations required more than external cloud controllers. Kubernetes also needed generic solutions for control-plane networking, image credentials and existing storage APIs.

Moving provider-specific code out of Kubernetes required replacements for behavior scattered across the control plane and nodes. The May 20 migration report describes four subsystems needed to preserve those capabilities. Cloud controller managers are the most visible part, but image-pull credentials and control-plane network access also depended on cloud knowledge. Existing storage objects created a further compatibility problem because their APIs had to remain usable after implementation code moved elsewhere.

The report announces that cloud-provider integrations have migrated from the core repository to external plugins. Its authors estimate roughly 1.5 million lines removed and approximately 40 percent smaller core binaries. Those are project-reported results of the migration, not measurements of a particular installation. The same report says that permanent disabling and removal of in-tree providers begins with the forthcoming 1.31 release. The milestone describes completed migration work ahead of that release boundary.

I see the significance in the replacement contracts. Kubernetes's value as an extensible controller system depends on keeping infrastructure-specific behavior outside the parts that every cluster must carry. Removing code is useful, but preserving the capabilities through explicit interfaces is the harder part of this change.

Node metadata and load balancers

The cloud controller manager replaces cloud interactions formerly located in kube-controller-manager and kubelet. The report gives node initialization as a concrete responsibility. A provider knows information such as a node's region and zone, and may know addresses unavailable through local inspection alone. The external controller applies the corresponding metadata to the Node resource.

That relationship matters because removing built-in cloud code does not make infrastructure facts irrelevant. Kubernetes still needs the metadata, but the provider implementation now supplies it through a separately operated component. The architectural change relocates responsibility for obtaining and applying that information. It also makes the provider-specific dependency more explicit in the cluster's configuration.

The same component runs the service controller that provisions cloud load balancers for Services of type LoadBalancer. A Service remains the Kubernetes-side declaration, while the controller translates that declaration into actions against the provider. The external implementation still has to maintain the correspondence between the API object and the cloud resource. Extraction changes where that reconciliation lives, without making the provider API part of the portable Service definition.

The 1.29 migration guidance explains how the transition becomes visible to operators. Kubernetes 1.29 disables the legacy integrations by default. The guidance identifies DisableCloudProviders and DisableKubeletCloudCredentialProviders as the relevant gates. Opting back into old behavior is a temporary compatibility measure, with eventual removal already planned in that earlier announcement.

For an existing installation, external mode therefore requires an actual provider controller. A flag expressing the new integration arrangement does not itself supply the missing implementation. This is the operational counterpart to separating the code, the cluster has to deploy and maintain the component that owns the provider interaction.

Control-plane networking without provider knowledge

The API server's old SSH tunneler depended on provider-specific details to establish connections between the control plane and nodes. That is a different use of cloud integration from reconciling a Service. Replacing the cloud controller alone would leave this dependency inside another core component.

The migration report identifies the API Server Network Proxy as the replacement extension point. Initiated in 2018 with SIG API Machinery, it provides generic proxying for traffic from the API server to nodes. The report describes the extension point as generally available and connects it to the Konnectivity project. Its purpose is to let the API server use a secure proxy without understanding which cloud provides the machines.

This is a useful example of preserving behavior while changing the information a component needs. The API server still needs a working communication path. It no longer needs the provider-specific implementation used to construct the old SSH tunnels. A generic proxy contract allows the connection problem to be solved outside the API server's cloud-specific code.

The practical dependency remains real. An installation that requires this proxying must have the replacement path configured and functioning. Removing an embedded implementation does not remove the network relationship it served. The report's four-subsystem account is valuable because it prevents the migration from being mistaken for a single controller deployment.

Image credentials are another integration

The kubelet previously contained logic for obtaining credentials for registries hosted by Google Cloud, AWS and Azure. These credentials are described in the migration report as short-lived tokens used to pull images. That convenience depended on the kubelet knowing those environments and their APIs. It placed another provider-specific responsibility in a component deployed on cluster nodes.

Credential provider plugins replace that built-in behavior with an execution-based extension point. The kubelet invokes plugin binaries that provide credentials dynamically. Introduced in 2019, the mechanism makes the credential-fetching implementation external while retaining a generic way for the kubelet to request the information it needs.

There is also an extensibility consequence beyond removal of the original providers. The report says the mechanism allows short-lived token retrieval beyond those initial three environments. A generic extension point can support providers that were never compiled into Kubernetes. That makes consistency of integration a practical benefit, because new implementations can use the same arrangement as the original ones.

This portion of the migration is easy to overlook when a cluster's visible cloud resources already work. A correctly initialized Node and an available load balancer do not establish that the node can obtain credentials for its next image pull. The systems satisfy different contracts. Their configuration and ownership need to be understood separately even when they rely on the same cloud account infrastructure.

I prefer that explicit separation of concerns. It makes it possible to discuss image authentication as a node dependency with a specific plugin implementation. The kubelet can remain responsible for requesting credentials without acquiring knowledge of every provider's authentication API.

Preserving storage APIs during implementation removal

Storage required a different migration technique because existing resources already used the built-in volume APIs. Simply replacing those APIs with an external driver interface would leave existing objects with no implementation. The report describes a translation layer that converts the in-tree volume API into the equivalent CSI API and redirects storage operations to the CSI driver.

That layer preserves API compatibility while allowing implementation code to leave the core repository. It separates the user-facing representation from the code carrying out the operation. Existing resource definitions can remain meaningful even though an external driver now performs the storage work. This is a more demanding compatibility obligation than merely documenting a new way to create future volumes.

The report describes CSI drivers as pods that communicate with kube-controller-manager's storage controllers through the Kubernetes API and with kubelet through a local gRPC endpoint. Those distinct communication paths reflect the different places where storage operations need integration. Externalizing storage still requires cooperation between the control plane and the node where the workload consumes the volume.

The CSI migration also explains why counting deleted lines is an incomplete description of the engineering. Removing the old volume implementation depends on both a replacement driver contract and a compatibility path for existing APIs. Without the translation layer, code removal would impose a different and more disruptive operation on users of those objects.

Independent releases still need coordination

The earlier 1.29 guidance states the motivation in terms of provider implementations developing and releasing independently of Kubernetes core. It also describes consistent integration as an objective. These goals reinforce each other when the external contract is clear, but they still require maintenance across projects. An external controller or plugin is another installed dependency with its own lifecycle.

The May report recognizes that testing remains part of the work ahead. SIG Cloud Provider plans broader controller testing and a framework covering more providers. That follows directly from the architecture, shared interfaces need evidence that implementations produce the expected behavior. Separating repositories creates a stronger need to make those expectations testable across independently maintained components.

The immediate migration work therefore spans the cluster's actual uses of cloud integration. Node metadata and load-balancer reconciliation belong to the external controller, while proxying and image credentials have their own extension points. Existing storage APIs rely on translation to CSI operations. The report makes the completed code migration understandable by identifying those replacement systems and the compatibility responsibilities each one carries. The forthcoming release boundary is consequently the end of a transition already visible in earlier component defaults, with explicit replacement mechanisms available for the affected capabilities.