Kubernetes 1.37 stabilizes resource metrics and storage migration

Kubernetes 1.37 promotes the resource metrics API and storage-version migration to stable. These changes concern different parts of cluster operation, but both require care with version transitions. A stable API contract does not establish that an optional provider serves it, and a new storage version does not establish that existing objects have been rewritten.

I am particularly interested in these changes because Kubernetes API machinery is the part of the system I value most. After eight years with Kubernetes, including Talos clusters with Cilium and GitOps, I want declared configuration to have an observable operational result. API stability helps define the contract. An upgrade procedure still needs evidence about the components serving that contract and the data being migrated.

The release also promotes HPA scale-to-zero support to beta and enables it by default. That feature uses object or external metrics, while the stable resource metrics API covers CPU and memory. Keeping those metrics categories distinct matters when planning adoption. The presence of a stable CPU and memory API cannot supply a demand signal for a workload with no running pods.

Resource metrics finally has a stable version

The release announces metrics.k8s.io/v1 after a long beta period. The metrics API enhancement proposal explains the intent. Its stable API keeps the same structure as v1beta1, with the version name changed. The goal is a nonbreaking graduation, without removing beta support as part of the transition.

This is an API maturity change. It does not introduce a new category of telemetry or turn the API into a general-purpose monitoring system. The proposal describes resource metrics for nodes and pods, with container usage included in pod metrics. HPA and kubectl top are established consumers of this information. I would keep the scope clear when discussing what the graduation provides to an application platform.

The proposal records the API's alpha introduction in Kubernetes 1.6 and beta promotion in 1.8. It explains that the long-running beta state predates the formal enhancement process and conflicts with the project's effort to avoid permanent beta APIs. Stability makes the compatibility commitment explicit for an interface whose users have already relied on it for years.

I would not infer a change in metric values from the version name. The proposal makes changing the schema a non-goal. Adoption testing should verify that consumers continue receiving the expected resource measurements through the available version. A change in measured usage would need investigation of the collection and serving path, with evidence beyond the graduation itself.

Aggregation requires an exact served version

Resource metrics arrive through the API aggregation layer. The proposal describes a requester sending a metrics request through the Kubernetes API server to an add-on API server such as metrics-server. An APIService registration connects a particular group and version to that provider. This is a distinct component relationship to check during an upgrade.

The proposal explicitly warns that aggregation does not provide the version conversion behavior an operator might expect from other APIs. A request for v1 needs an available v1 APIService registration. A beta-only registration does not automatically serve it. Both registrations can exist during migration, and requests for each version then use the matching registration.

For a named test context, I would inspect the available APIService objects before assuming the new endpoint exists:

kubectl --context staging get apiservices

The context is illustrative. This is an inventory check, not a complete validation of the metrics path. The next check should exercise the intended consumer under its real permissions and verify that the provider returns useful data. A registered service and a successful application-level metrics request provide different evidence.

The enhancement proposal describes discovery and fallback to v1beta1 for in-tree consumers when v1 is unavailable. I would still review independently developed clients. A client that hardcodes an endpoint has a different compatibility obligation from one that discovers the served version. The control-plane upgrade should not become an assumption that every metrics client has changed at the same time.

I would introduce stable-version support on the provider side while preserving beta access for existing clients. The release says beta remains usable during the transition under the deprecation policy. Any plan to remove the old registration needs its own consumer inventory. Graduation provides a migration path without requiring an immediate deletion of the previous version.

Storage migration rewrites existing objects

The release also stabilizes storagemigration.k8s.io/v1 and enables it by default. A StorageVersionMigration object asks the built-in controller to migrate existing resources to the default storage version for that API. This applies to built-in resources and custom resources. The announcement also identifies rewriting data after encryption-at-rest changes as a use case.

The storage-version migration proposal explains why this needs an explicit process. Changing which version the API stores does not by itself rewrite every existing object. The migration controller supplies the rewriting work that administrators previously handled through scripts or a separately deployed migrator.

The proposal separates the migration controller from automatic triggering. It describes moving the migrator into kube-controller-manager while leaving creation of the migration object to the user. I would preserve that distinction in the upgrade procedure. An enabled controller is ready to perform requested work, but the operator still needs a reason to request a particular resource migration.

For a custom API upgrade, I would identify the target storage version and establish when rewriting is safe. The migration request belongs with the upgrade record, including its observed result. A procedure that changes the CRD and then immediately removes old-version support needs evidence that the stored data no longer depends on the old version.

The controller accounts for concurrent changes

The proposal describes using the garbage collector's object cache as the migration inventory. The controller first waits for that cache to reach an observed resource version. It then issues minimal patch requests for eligible objects. Including the object's UID protects against accidentally recreating an object that disappeared during the migration.

Resource-version checks address another concurrent-change case. If someone has already modified an object since the migration's observed point, a conflict means the migration need not rewrite that older state. The operation has to cooperate with ordinary cluster activity. I would assess its completion through controller status and the documented upgrade requirements instead of treating it as an isolated database script.

For custom resources, the proposal also discusses updating stored-version information after migration. It guards that update against CRD changes during the operation. I would keep changes to the API definition controlled while evaluating the migration result. Otherwise the operator risks drawing conclusions from a completion state that concerns a different generation of the custom-resource definition.

Encryption-at-rest rotation needs the same distinction between new writes and existing data. The release says the migrator can rewrite resources under new encryption settings. I would make the encryption configuration change and the rewrite request separately observable in the procedure. Successful new writes do not establish that older stored objects have passed through the new configuration.

I would test migration with a custom resource that has existing instances before the storage-version change. The exercise should include an instance updated during migration and an instance deleted during migration. Those cases correspond to the concurrency protections described in the proposal. The review should explain what status constitutes completion and what an operator should inspect if the migration does not finish. This would give the production procedure a tested recovery path.

Plan the upgrade around the actual consumers

For the metrics API, I would inventory providers and clients, then verify which exact versions they serve and request. For storage migration, I would inventory the resource types requiring a rewrite and define the completion checks before issuing migration requests. These are different operational tasks even though both use Kubernetes API versioning.

I would evaluate scale-to-zero separately as well. The release requires object or external metrics and describes a ScaledToZero status condition that distinguishes HPA-controlled idling from manual deactivation. That lifecycle needs tests of returning demand. It should not be included automatically in a resource-metrics version transition merely because both changes involve autoscaling.

The stable APIs give operators clearer contracts for these procedures. I would use that stability to simplify future upgrade documentation, while keeping evidence about the running provider and completed rewrites in the change record. The next reviewer should be able to establish which metrics versions remain in use and which resource migrations actually finished.