March 13, 2024·6 min read·engineeringistioupgradestls

Istio 1.21 separates upgrade timing from behavior changes

Compatibility versions let an Istio 1.21 installation retain selected 1.20 behavior. The first uses cover ExternalName handling and TLS origination defaults, where small changes can alter live traffic.

A software upgrade can be necessary even when the associated behavior changes need a different deployment schedule. Istio 1.21 introduces compatibility versions to separate those decisions for selected changes. The March 13 announcement describes the problem in operational terms, small corrections can disrupt production traffic and discourage users from upgrading. Retaining an older default temporarily gives those corrections a transition path without requiring the entire installation to remain on the older binary.

The first release using this mechanism covers ExternalName service handling, automatic SNI for TLS origination and default certificate verification for TLS origination. These are behavior changes with concrete consequences for destination selection and trust. They are not merely formatting changes in configuration output. An existing object can acquire a different effect under the new defaults even when its manifest remains unchanged.

I see this as a useful response to the cost of operating Istio as another distributed system. Versioned configuration is valuable only when the relationship between configuration and proxy behavior remains understandable. Compatibility versions make selected default transitions explicit, so an implementation upgrade and adoption of a new default can be reasoned about separately.

Compatibility is selected during installation

The announcement gives the installation setting --set compatibilityVersion=1.20 for retaining the selected 1.20 behavior while running Istio 1.21. The setting describes a compatibility target. It does not change the installed software back into the older release. That distinction is the purpose of the mechanism, an operator can move the implementation forward while postponing particular behavioral changes covered by compatibility handling.

The scope remains specific. The announcement lists the changes gated in this release, rather than promising that every possible observable difference disappears. A compatibility version should therefore be read together with the release and upgrade notes. It is a controlled way to retain selected defaults, not a claim that the new release is indistinguishable from its predecessor.

Istio also adds precheck support for identifying potentially affected resources. The announcement describes istioctl experimental precheck with a source-version argument and shows a diagnostic identifying a changed SNI default. The diagnostic connects the resource to the behavior change and suggests the compatibility setting. That is more useful than requiring administrators to infer impact from a list of feature-flag names alone.

This makes the source version part of the analysis. The question is which behaviors change relative to the installation being upgraded. A report of potential impact is still distinct from proof that application traffic will fail. It identifies configuration deserving attention and gives the administrator a way to preserve the older default while resolving that dependency.

Automatic SNI changes TLS origination

The 1.21 upgrade notes say ENABLE_AUTO_SNI becomes true by default. When a DestinationRule does not explicitly set SNI, Istio sets it from the downstream HTTP host or authority header. The configuration therefore gains a value selected from request information where the previous default did not supply that same automatic behavior.

That change matters because the TLS connection to the upstream destination now includes a name derived from the request. An operator reading only an unchanged DestinationRule might miss the source of that value. The new default makes the behavior more automatic, while the compatibility setting provides an explicit path to retain the older arrangement.

The announcement narrows the first release's automatic-SNI change to SIMPLE TLS origination in DestinationRule. Keeping that scope visible avoids treating the release note as a statement about every TLS connection in the mesh. Workload-to-workload identity and externally originated TLS have related security concerns, but they are different configuration paths.

This is a suitable use of compatibility handling because a default can be desirable without being harmless for every existing deployment. The project can adopt the new behavior while acknowledging that current traffic may depend on what the old release omitted. The transition becomes part of the supported configuration model rather than an undocumented collection of exceptions.

Certificate verification becomes the default

The upgrade notes also set VERIFY_CERT_AT_CLIENT to true by default. When DestinationRule does not specify caCertificates, server certificates are verified using the operating system's CA certificates. The old absence of an explicit CA path therefore no longer means that the connection follows the previous verification behavior.

The notes describe retaining older behavior through compatibility versions and mention insecureSkipVerify as an explicit DestinationRule option. These choices have different descriptive value. A compatibility target retains the covered previous defaults during a transition. An explicit verification bypass records a security decision in the destination configuration itself. Neither should be mistaken for installing additional trusted certificate authorities.

The operating-system trust source is an important part of the mechanism. Default verification is useful because it supplies a trust check without requiring an explicit CA file in every rule. It also makes the available OS CA certificates relevant to whether a connection succeeds. A destination certificate must satisfy the actual configured verification behavior, not merely complete a TLS handshake.

I favor making verification the normal behavior while keeping its transition visible. A mesh's identity features can encourage an overly broad sense that encryption alone establishes the intended peer. These upgrade notes instead identify exactly which connection path changes its certificate checking and which configuration supplied, or failed to supply, the trust material.

ExternalName becomes an alias

The ExternalName change addresses a different mismatch. Kubernetes implements an ExternalName Service through a DNS CNAME relationship. Istio historically treated each such Service as its own service and sent matching traffic to its configured DNS name. The upgrade notes list several problems caused by that additional interpretation.

Ports were required by Istio even though Kubernetes did not require them for this use. Non-HTTP port handling could match all traffic on a port and direct it to the wrong place. Treating the destination name as opaque also interfered with applying Istio policies to the actual destination. The notes give an in-cluster target without expected mTLS as an example of that policy mismatch.

Istio 1.21 treats ExternalName Services as aliases. Where a concrete service hostname would match, its ExternalName alias can match too. Kubernetes DNS continues performing the CNAME behavior outside Istio. The mesh's interpretation now follows the relationship between the alias and the actual service instead of inventing an independent destination with its own policy identity.

This changes existing configuration semantics. The Service's ports field is no longer needed and has no effect. A VirtualService routing to an ExternalName needs the referenced service represented as a Service or ServiceEntry. DestinationRules need to apply to the referenced service rather than the alias. These requirements describe the actual target whose behavior and policy the mesh must understand.

The upgrade notes say the alias implementation appeared in the previous release but was disabled by default. In 1.21 it becomes the default, and the notes describe an environment-variable opt-out as well as compatibility handling. That history reinforces the distinction between code availability and behavior activation. The implementation can be present before an upgrade changes which path existing configuration uses.

Other release improvements have different scopes

The announcement reports sidecar images approximately 25 percent smaller and binaries roughly 10 MB smaller. It also reports a typical 5 MB memory reduction associated with the binary change. Those are project-reported release results, not a guarantee for every workload. They matter because a sidecar image is deployed repeatedly across application pods, making per-proxy size relevant at fleet scale.

Ambient's in-pod traffic redirection also arrives with broader CNI compatibility. The release points to the January design work and reports tests across managed platforms and third-party CNIs. Ambient is still targeting beta in a subsequent release. Improved compatibility in 1.21 should not be conflated with a maturity graduation that the announcement has not made.

The upgrade notes separately change the Gateway name label to gateway.networking.k8s.io/gateway-name. Existing generated Deployments and Services retain compatibility handling for their old selectors. Policies and scripts consuming the old label still need their own attention. This is another example of a shared configuration dependency extending beyond the component that originally created the label.

For compatibility versions, the immediate contribution is narrower and concrete. Selected TLS and service-alias defaults can advance without making adoption simultaneous with the binary upgrade. The affected resource diagnostics and upgrade notes expose what changes, while the compatibility setting provides a temporary prior behavior for the specific transitions it covers.