March 10, 2023·5 min read·engineeringkubernetesregistriesnetworking

The Kubernetes registry redirect changes the image pull path

On March 20, k8s.gcr.io will redirect clients to registry.k8s.io. The change exposes network and client assumptions even when an image reference stays the same.

Changing a default did not move most Kubernetes image traffic. The project switched its default registry with Kubernetes 1.25, but the March 10 redirect announcement says most pulls still reach k8s.gcr.io. That leaves donated hosting resources at other providers underused while the old delivery path continues to incur costs. The authors describe a risk of running out of funds to serve that traffic.

On March 20, the old registry will start redirecting traffic to registry.k8s.io. This changes where clients retrieve images even when their manifests still name the old endpoint. For users whose runtimes and networks accept the redirect, that is a migration aid. For a network restricted to the original domain, it changes the effective dependency without changing the configuration text.

I like declarative systems because their maintained configuration makes intent inspectable. A registry redirect is a useful reminder that an external service can change the execution of that intent. The hostname in a manifest identifies the start of a retrieval process, and it need not identify every server involved in supplying the image.

Distribution costs forced a more active migration

The project already explained its new registry architecture in the November distribution announcement. registry.k8s.io distributes delivery across providers and regions through a blob redirector. This lets the community use hosting resources contributed by several providers and reduces dependence on the old custom Google Container Registry endpoint.

Asking consumers to change a hostname leaves migration dependent on every maintained manifest and installation process. The March announcement documents the result of that approach so far. Most traffic has remained at the old address, despite the new default. Redirecting existing requests allows the project to use its new distribution resources without waiting for every consumer to edit its configuration.

This does not make the old address a permanent compatibility guarantee. The announcement calls the redirect a stopgap and asks users to update their manifests. It says the deprecated endpoint will eventually be phased out, without supplying a final removal date. Keeping old references solely because they still resolve would turn a temporary migration mechanism into an undocumented long-term dependency.

There is also a separate content deadline. The February freeze announcement sets April 3 as the date after which the old registry will receive no new images. The March article repeats that releases, patches, and security updates will stop arriving there. A redirect changes request routing, while a freeze changes the publication policy for the old registry. Operators need to understand both dates.

Two independent assumptions can fail

A client must handle the HTTP redirect, and its network must permit the destination. The announcement identifies a small subset of non-standard clients that do not handle registry redirects. Those clients need a direct reference to registry.k8s.io. That fixes the dependency on the old endpoint's redirect behaviour, but it does not by itself resolve a restrictive network policy.

The new registry can direct delivery to backing infrastructure. Its architecture therefore does not fit an allowlist limited to k8s.gcr.io. The November article explains that backend domains and addresses are expected to change as more providers contribute resources. Allowing the initial request while denying a later destination leaves the image pull incomplete.

Reported symptoms depend on the runtime and selected endpoint. The March article lists ErrImagePull, ImagePullBackOff, and FailedCreatePodSandBox among possible failures. It also gives a certificate-trust failure against a backing host as an example. These errors identify failed retrieval or setup, but they do not all establish the same cause. A redirect-incompatible client and a proxy that rejects a certificate need different corrections.

The sandbox example broadens the investigation beyond an application's image. Before application startup, the runtime may need an image for the pod environment itself. Availability of an application image does not establish availability of that other dependency. A node that cannot create a sandbox can consequently fail before the application process has an opportunity to run.

The change includes subproject images

The announcement states that all images on k8s.gcr.io are affected. It names NodeLocal DNS cache, the ingress-nginx controller, and node-problem-detector as examples beyond Kubernetes release images. Treating this as a control-plane-only migration would miss infrastructure deployed as ordinary workloads.

Existing pods are one place to inspect references. Maintained charts and manifests are another, particularly for operators who do not have access to every cluster. The announcement recommends searching those definitions for the old hostname. A reference that creates no pod today can still become an image pull during an installation, replacement, or future deployment.

For a restricted environment, the project recommends copying required images into a private registry or configuring a pull-through cache. It names crane copy SRC DST as one available copy operation. The organisation then controls the endpoint exposed to nodes, while its image-supply process handles the upstream distribution service.

That arrangement adds an explicit responsibility for supplying the images required by the maintained definitions. Copying an image is not equivalent to updating the references that consume it. A repository with current content and a chart still pointing at k8s.gcr.io leave the original dependency in place. The preparation process needs to connect the published artifact to the consuming configuration.

Admission policy can expose remaining references

The March article points to Gatekeeper and Kyverno examples for blocking images that use the deprecated registry. It also describes mutation as a last possible stopgap until maintained manifests are updated. These approaches have different effects on configuration ownership. Rejection makes the remaining dependency visible to the submitting workflow. Mutation lets a request proceed after another component changes the image reference.

I prefer correcting the maintained definition when that is possible. An admission rewrite adds another place an engineer must inspect to explain why a running workload differs from its submitted configuration. It can provide a migration bridge, but the original reference still needs an owner and a replacement.

The announced redirect lets the community reduce distribution costs while consumers make those corrections. Its practical limit is precise. It can route an old request toward the new registry, but it cannot grant network access, repair client redirect handling, or change the manifests that will submit the next request.