September 2, 2020·4 min read·engineeringkubernetesnetworkingendpointslices

EndpointSlices reduce the cost of changing one backend

A large Endpoints object makes a small backend change expensive to distribute. EndpointSlices split that object into smaller update units and give consumers an explicit aggregation responsibility.

Replacing one pod behind a large Service can force Kubernetes to distribute information about every backend again. The old Endpoints representation places a Service's endpoints in one object. Every kube-proxy watching that object receives the updated whole, even when the change concerns a single address. EndpointSlices reduce the size of that update by dividing a Service's endpoints among multiple objects.

Rob Scott's September 2 explanation describes the API as enabled by default in Kubernetes 1.19, with kube-proxy reading EndpointSlices instead of Endpoints. The practical benefit comes from the relationship between object size and watch fan-out. Smaller update units reduce the amount of unchanged data sent to consumers during backend churn.

I consider this an important refinement of the Kubernetes watch model. Watches let controllers respond to current state, but the representation of that state determines how much information each update carries. A useful API needs to account for both the number of consumers and the size of the object they receive.

One object amplifies every change

Scott illustrates the old cost with a Service backed by five thousand pods and a hypothetical 1.5 MB Endpoints object. If three thousand nodes each watch that object, one update distributes about 4.5 GB of data across the cluster. The example is arithmetic for that configuration, not a measurement that applies to every Service.

The example also explains why rolling replacement is expensive. Replacing many pods produces many endpoint changes, and each change can repeat the same large distribution. The total depends on how updates occur, but the structural problem remains even if the API server combines some changes. A representation containing every backend makes a small change large for every watcher.

Object size imposes another constraint. The article cites etcd's default 1.5 MB object limit and notes that some Endpoints resources can reach it at around five thousand pod IPs. Depending on the represented data, the precise count varies. The entire set must fit inside one stored resource.

Slices change the update unit

EndpointSlices store smaller groups of endpoints. The default controller target is up to one hundred endpoints per slice, configurable through --max-endpoints-per-slice on kube-controller-manager. Adding or removing a pod can then update a small slice instead of the full Service-wide list.

The discovery API definitions shipped with v1.19.0 describe a slice as a subset of the endpoints implementing a Service. Consumers select related slices through labels and join them to obtain the complete set. That aggregation obligation is part of the change in data model.

The distinction matters for a client written around a single Endpoints object. Reading one EndpointSlice is not equivalent to reading the full backend set. A consumer has to handle multiple objects and keep its combined view consistent as they change. The scalability benefit comes with that more distributed representation of the same logical set.

The source definitions allow up to one thousand endpoints in an individual slice. That API limit differs from the controller's default target of one hundred. Confusing the two would misstate the default behavior and the object's permitted size. The default is an implementation choice about grouping within the larger API constraint.

Endpoint information includes readiness and address type

A slice records addresses and ports, with endpoint readiness and topology information. Its addressType describes the kind of address carried by the slice, and all addresses in that slice have the same type. The field is immutable after creation. The API supports distinct IPv4 and IPv6 types, alongside FQDN addresses.

Readiness is represented as an optional boolean. The API comments describe a missing value as unknown and say consumers should generally interpret that state as ready. Clients therefore need to follow the defined semantics rather than treating an absent boolean as an automatic rejection. Small details in optional fields can affect the resulting backend set.

The topology map can identify the endpoint's host, zone, or region. Scott describes this information as a basis for future routing work. It is useful to distinguish information stored in an API from behavior already implemented by every consumer. Carrying a zone does not by itself establish that all traffic stays within that zone.

Existing consumers have a migration decision

The article states that the Endpoints API remains generally available and stable. EndpointSlices do not require every client to switch immediately in order to preserve that status. They do provide the representation on which additional functionality can be built, so direct consumers of Endpoints have a reason to consider supporting them.

The discussion describes dual-stack Services and topology-aware routing as work enabled by the new representation. Endpoint subsetting is an exploration that could let a proxy watch fewer slices, perhaps those containing endpoints in its zone. Those are stated directions at this date. They should not be mistaken for a promise that the default 1.19 setup already implements all of them.

My preference is to evaluate the current benefit independently of that future work. Splitting backend state reduces the data attached to a small change and avoids requiring every address to fit into one object. Clients gain that benefit when they consume the complete slice set correctly. Their aggregation logic becomes the place that translates multiple API resources back into the Service-wide backend view they need.