Gateway API adds inference-aware endpoint selection

A model server can be reachable and still be a poor destination for the next request. Its accelerator may already hold active inference sessions, its queue may be long, or it may lack the adapter needed by the requested model. A routing rule that sees only an HTTP path and a collection of healthy endpoints cannot express all of those differences.

The June 5 introduction to Gateway API Inference Extension describes a way to add those decisions to Kubernetes gateways. It preserves Gateways and HTTPRoutes while introducing inference-specific resources and an endpoint selection extension. The gateway identifies the appropriate pool, then asks the extension which model-server Pod should receive the request.

I like Kubernetes most where its API lets a domain-specific controller express something the built-in objects cannot. Inference routing fits that pattern. The useful addition is a declared relationship between the model a client requests and the capacity capable of serving it, together with a mechanism for selecting that capacity.

Model identity and deployment capacity have different owners

The introduction defines InferencePool as a set of model-server Pods on shared compute. It is comparable to a Service in its role of grouping endpoints, but it adds the information needed for inference-aware selection. A platform operator can manage the pool without requiring every client to know the names or addresses of its individual Pods.

InferenceModel represents the user-facing model endpoint. It maps a public model name to an actual model within a pool and supports policy such as traffic splitting or prioritization. This lets the name a client uses remain separate from the deployment details that satisfy the request.

That separation is consequential for ownership. The people responsible for serving capacity can change its placement and operation. Model owners can control the model name and rollout policy. Encoding both concerns as one opaque backend address would make either change depend on knowledge held by the other group.

The API names here describe the design published today. This is an evolving extension, so an example needs to identify the version it targets. The v0.3.0 repository documentation still describes the project as alpha and unsuitable for production. Publication on the Kubernetes blog does not promote the entire extension to a stable Kubernetes capability.

Selection needs information the gateway does not otherwise have

The request flow begins with ordinary gateway routing. An HTTPRoute matches a request and identifies an InferencePool backend. The endpoint selection extension then examines model-server information such as queue lengths, memory use and loaded adapters. The gateway forwards to the selected endpoint.

The extension changes the unit of the decision. Healthy endpoints can differ in their suitability for a particular inference request. Looking at real-time serving state allows the selector to account for that difference before the gateway commits the request to a Pod.

Long-lived inference requests make this useful because equal request counts need not imply equal current work. The introduction describes sessions that retain in-memory token caches and occupy accelerator resources. Model identity and request criticality add further information that ordinary path matching does not contain.

The repository documentation describes integration with proxies and gateways that support external processing. That is an implementation requirement, not something any HTTP proxy acquires merely by installing the custom resources. The gateway must participate in the selection protocol and use the chosen endpoint.

The early implementation also requires a model server that exposes the necessary metrics. The v0.3.0 documentation identifies vLLM with the relevant metric support and describes other server integrations as forthcoming. A standardized API does not remove the need for a concrete protocol between the selector and the server reporting its state.

The early pool schema exposes a failure decision

The v0.3.0 InferencePool schema is useful because it makes the operational contract specific. It requires a selector, targetPortNumber and extensionRef. The selector matches model-server Pods in the pool's namespace. Cross-namespace selection is not supported by that schema.

This small example targets its v1alpha2 API. It assumes the custom resource definition is installed, that a compatible gateway references the pool, and that the named endpoint-picker Service exists. The labels and names are illustrative deployment choices.

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata:
  name: text-models
spec:
  selector:
    app: text-server
  targetPortNumber: 8000
  extensionRef:
    name: endpoint-picker
    failureMode: FailClose

The extension reference includes a failureMode with FailOpen and FailClose as the allowed values. FailClose is the default in this schema. This makes an unavailable endpoint picker an explicit gateway behavior decision. Treating the picker as a minor optimization would overlook the possibility that inference-specific selection has become a required step in admitting a request.

I consider that field as important as the selection policy. Adding a service to the request path adds its availability behavior to the serving contract. The API needs to state what happens when the service cannot answer, even if its normal answer improves utilization.

The published benchmark supports a narrower result

The introduction compares the extension with a standard Kubernetes Service using a vLLM deployment with ten Llama2 replicas on H100-backed Pods. Its traffic generator uses ShareGPT-derived requests and increases offered traffic. These details describe the experiment, without establishing a universal result for every model and accelerator.

The reported throughput is roughly comparable across the tested range. Tail latency improves at higher offered loads, including the per-output-token and overall p90 measurements discussed in the post. The useful interpretation is that informed endpoint choice can avoid concentrating work where queues and memory pressure are already high.

That result does not show that routing creates additional accelerator capacity. It shows a change in how requests use the capacity in that deployment. A workload with different request lengths or serving behavior can expose a different balance between queueing, cache reuse and available endpoints.

The roadmap also separates intended capabilities from current ones. It includes remote prefix-cache awareness and fairness within a criticality band, as well as aggregate metrics for autoscaling. Those are future directions in today's introduction. They should not be read as promises already delivered by every implementation of the early API.

The immediate architectural contribution is a defined point where inference-serving knowledge participates in an existing gateway request. HTTPRoute still describes routing, the pool names eligible serving capacity, and the extension chooses a destination using information about that capacity. That gives model-aware selection a place in the Kubernetes configuration model while leaving its protocol and failure behavior explicit.