Istio 1.0.4 corrects stale endpoint removal

A service can have healthy replacement instances while a proxy still holds destinations which should have disappeared. The Istio 1.0.4 release on November 21 corrects missing removal of stale endpoints which caused HTTP 503 errors. The failure concerns the mesh's view of service membership. Adding a new endpoint does not compensate for retaining an obsolete destination in the set used for forwarding.

I regard endpoint convergence as a requirement for trusting a mesh with application traffic. Workloads already change as an orchestrator schedules replacements. A traffic-management system must reflect those changes in both directions. Retaining an old address can turn normal workload churn into application failures even when the service has adequate healthy capacity.

Endpoint membership is assembled for a particular destination

The 1.0.4 endpoint discovery implementation describes EDS as the endpoint set associated with a service or a label-selected subset. In Kubernetes, endpoint changes typically follow readiness decisions. For subsets, the control plane also needs the workload labels which determine whether an instance belongs in the selected version.

This means an Envoy destination is more specific than a list of every pod with an address. The code parses the cluster name to obtain the hostname, port and subset, then resolves the subset's labels. It filters endpoint entries by the service port name and matching labels before constructing a load assignment. Different routes to one service can consequently depend on different membership sets.

That specificity affects removal as well as addition. An instance can leave a subset because its labels change while the broader service still exists. A port mapping can also affect which entries belong in an assignment. The control plane has to regenerate the destination set which the particular route uses, not merely keep a global inventory of live machines.

Multicluster membership uses independent shards

The implementation keeps endpoint information in shards grouped under each service. Its comments explain that individual Kubernetes clusters or registries can update their shard independently. An update for one cluster contains that cluster's full endpoint list for the service. Those shard lists are later filtered and combined into the assignment used by a proxy.

This organization avoids treating one cluster's update as the entire multicluster service. It also makes replacement semantics important. If a shard's latest list no longer includes an endpoint, keeping the old entry would misrepresent the source cluster's current membership. A merged assignment is only as accurate as the shard state from which it is assembled.

The incremental update path builds a new locality grouping from the current shard entries. It then replaces the stored load assignment under the cluster's lock. When no matching instances remain, the code records a no-instances diagnostic and sets the instance metric to zero. An empty result is meaningful state which has to reach the forwarding system, rather than being mistaken for a lack of information to update. The assignment still names its destination cluster even when the resulting endpoint list is empty.

The same file includes a full-update path used when incremental information is unavailable. Missing service-port information or a missing shard set can trigger that fallback. This separates the optimization for known changed services from the ability to rebuild an assignment through service discovery. The code documents the mechanism without establishing that every HTTP 503 has the same cause as the stale-endpoint defect.

Label changes can require broader regeneration

Workload updates maintain labels and annotations by workload identity. When labels do not change, this path avoids a push. A changed label set instead requests a full configuration update. The source comments favor that broader operation because a narrower update might miss other affected workloads.

I find this a useful example of the correctness tradeoff in incremental discovery. Narrow updates reduce repeated work, but membership relationships can span several generated configurations. The implementation has to recognize when its local change information is insufficient to establish the complete affected set. A full update preserves a more conservative path for those cases.

The release also documents a separate operational problem with proxy synchronization inspection. It warns that istioctl proxy-status can deadlock Pilot and recommends avoiding the command. The reported failure includes continued memory growth after the deadlock. This is distinct from stale endpoint removal and should not be interpreted as a diagnostic step which is safe merely because it is read-oriented.

The notes include fixes for sidecar injection with slash-containing pod labels and data corruption in out-of-process Mixer adapters. Those affect different representations in the mesh. For endpoint failures, the relevant state remains the service's shard membership and the load assignment generated for the requested port and subset, including destinations which must now be absent.