Kubernetes 1.23.4 repairs updates to older Service objects

An old Kubernetes object can be valid when stored and still expose a defect years later when a new API server tries to update it. Kubernetes 1.23.4 contains a particularly clear example. Services that had not been modified since 1.19 could encounter an incorrect error requiring spec.clusterIPs. An operator could reach that error while updating an existing Service, without intending to change its networking configuration.

The 1.23.4 release, published on February 17, includes a fix for this regression. The interesting part is the update path's treatment of the old object. An API compatibility promise has to cover persisted representations, including fields that did not exist when the object entered storage. Accepting a fresh manifest under the current server is only part of that promise.

I value Kubernetes chiefly for its API machinery and reconciliation model. This patch shows why that machinery needs compatibility work beneath the declarative interface. A controller can keep submitting the desired object correctly while the server mishandles the historical state against which it compares the request.

The old object needs defaulting too

The backport in pull request 107875 identifies the affected case as a 1.23 regression involving previously persisted Services. Its code adds a call to defaultOnReadService(oldSvc) at the start of the relevant update processing. The accompanying comment explains that the saved object can predate fields the code expects to be defaulted.

That location matters. An update involves both the requested object and the previous stored object. Processing only the incoming representation does not establish that the previous representation contains every field the current comparison logic expects. The patch prepares the old Service before passing the pair into the logic that preserves allocated values.

The defect therefore does not imply that every Service needs a newly supplied list in its manifest. It means the server failed to normalize historical state before using it. Treating the validation message as proof that a user's configuration is incomplete would confuse an implementation regression with an application requirement.

The code change is small because the necessary defaulting function already exists. The missing operation belongs to one route through the API server. That is a common source of compatibility defects in software with multiple read and write paths. A function can be correct wherever it runs and still fail to protect a path that never calls it.

A test that represents an actual upgrade

The regression test in the same pull request stores a Service directly into etcd with missing fields that represent an older version. Its specimen contains a singular clusterIP and a port, without the newer plural field. The test then uses the client to update the Service through the running API server.

That construction is essential to the evidence. Creating a Service through the current API server could populate fields before the test reaches the update operation. Such a test would exercise a newly normalized object and miss the historical representation. Direct storage makes the old state an explicit input.

The relevant part of that representation can be illustrated as follows. This is a reduced description of the test's networking fields, not a proposed manual repair to a live Service.

spec:
  clusterIP: 10.0.0.1
  ports:
    - name: test-port
      port: 81

The test does not need a real cluster that has remained untouched since 1.19. It needs the persisted conditions produced by that history. Reconstructing those conditions makes the regression repeatable and keeps the test focused on the compatibility boundary.

This distinction is useful for people writing Kubernetes extensions as well. If an API evolves, tests that create all fixtures through its latest validation and defaulting code can conceal old-state defects. A migration test needs an input that preserves the older omission or representation.

Other fixes in the patch

The release also includes pull request 107761, a backport addressing a kubelet pod-status regression with static pods. Its release note describes unnecessary restarts in certain add and remove cases. The limited wording matters. It does not establish that all static pod restarts share this cause, or that static pod lifecycle handling has been redesigned.

A separate Azure File CSI migration fix addresses volume identifier collisions. That belongs to another compatibility boundary, where one storage integration translates identifiers for another. Together with the Service repair, it shows that maintenance work frequently concerns the preservation of meaning across implementation changes.

The patch also corrects OpenAPI serialization of x-kubernetes-validations. These are separate fixes with separate affected paths. Their presence in one patch release does not make them one failure mode.

My preference is to describe a patch through the behaviour it restores. In this case, an older Service should remain updatable under the newer API server without a false requirement for a field the server knows how to derive. The added defaulting call and historical-state test provide a precise explanation of how 1.23.4 restores that behaviour.