January 13, 2021·4 min read·engineeringkubernetescontrollersnetworking

Kubernetes 1.20.2 fixes resource leaks during load balancer reconciliation

A GCE load balancer fix in Kubernetes 1.20.2 shows why controllers need explicit cleanup on failed attempts. A quota correction addresses another boundary between declared intent and resource accounting.

A controller can retry a failed operation and still consume resources on every attempt. Kubernetes 1.20.2 addresses a concrete example in the GCE internal load balancer controller. A temporary IP reservation survived when load balancer creation failed, even though the reservation was only needed while the controller assembled the load balancer. Retrying the Service did not make that abandoned reservation useful.

The 1.20.2 release, published on 13 January, includes the correction. The interesting part of this maintenance release is the relationship between reconciliation and cleanup. A controller needs to converge the requested configuration, but its intermediate operations also need defined lifetimes. An unsuccessful attempt must leave external resources in a state the next attempt can handle.

I value Kubernetes for its reconciliation model. That model makes failure recoverable when controllers correctly account for the work they have already performed. This patch is a useful example of the implementation details behind that expectation.

The reservation outlived its purpose

The GCE fix moves address release into a deferred function immediately after the controller reserves the address. Previously, the release call appeared later in the successful creation path. A return caused by an intervening error skipped that cleanup.

The temporary reservation protects the address while the controller creates the resources that will use it. The patch shows work on firewall rules and forwarding rules after reservation. Those operations can fail. Under the previous arrangement, the controller reached the release call only after enough of that work succeeded. A Service that repeatedly encountered a creation error could therefore retain a cloud resource unrelated to any successful forwarding rule.

The corrected structure associates cleanup with acquisition. Once the reservation exists, the controller schedules its release before proceeding to the remaining operations. Go runs the deferred function when the enclosing function returns, including error returns. The same release path now covers successful completion and unsuccessful attempts.

This does not make the cloud operations transactional. The patch still logs an error if releasing the reservation itself fails, and its diagnostic explicitly acknowledges the possibility of an orphan. The improvement closes a specific control-flow gap. It does not establish that every failure in a distributed cloud API can be rolled back automatically.

That distinction matters when reading a maintenance note that says an IP leak is fixed. The repaired behavior concerns the lifetime of a temporary address reservation in this controller. It is not a promise that every address associated with every GCE load balancer disappears whenever a Service operation fails.

The tests describe the intended ownership

The regression tests check the address reservation after deliberately unsuccessful load balancer operations. They require the address lookup to return an error and no address. This tests the external state left by the attempt, which is the part that ordinary retry logic cannot infer from a function return value alone.

The patch also checks reservation release in successful resource assertions and after deletion assertions. Together, those checks define the reservation as temporary. Its lifetime ends when the ensure function exits, even when the resulting load balancer continues to use the selected address through its forwarding rule.

There is a useful engineering distinction here. A forwarding rule needs an address, while a temporary reservation coordinates its creation. Releasing the reservation is part of completing the setup operation. Treating both as resources with the same lifetime was the underlying mistake exposed by the error path.

For controller authors, this suggests a concrete review question. After a resource has been acquired, which later returns can skip its release? The answer needs to include errors from unrelated setup steps. A cleanup call placed near the end of a function can look correct while covering only its successful path.

The release also fixes Service quota checks. The quota patch describes a failure during ClusterIP creation involving a NodePort quota check. Its implementation removes zero-valued entries from the calculated resource usage delta before admission continues.

Previously, the code checked whether the entire delta was zero. That could short-circuit a request whose resource list contained only zeros, but it left zero-valued entries inside a mixed delta. The new helper builds a resource list containing only nonzero values. Admission returns early if that filtered list is empty.

The distinction is visible in the accompanying tests. An input containing zero CPU and memory alongside nonzero storage retains only storage. An all-zero input becomes empty. A list whose entries are already nonzero remains equivalent to its original value. These cases establish how the helper treats resource accounting independently of the Service that exposed the bug.

A zero entry and an absent entry can have different consequences when later code chooses which quotas to evaluate. Filtering the delta makes the next stage operate on resources whose usage actually changed. The patch therefore corrects the representation passed between admission stages, without changing what a ClusterIP Service is intended to request.

Maintenance of the reconciliation contract

The 1.20 changelog also lists Azure file cleanup and latency fixes, a CPUManager map-access race correction, and restoration of missing cAdvisor machine metrics. These are separate fixes, with different affected components. The GCE change is especially informative because the resource lifetime and its failed path are both visible in a small patch.

My interest in Kubernetes is strongest where its API machinery makes operational behavior explicit. Here the useful contract extends beyond accepting a Service object. A reconciliation attempt must release the temporary reservation it acquired, and quota admission must carry forward an accurate usage delta. Kubernetes 1.20.2 repairs those two implementation boundaries while retaining the existing Service model.