Kubernetes 1.9.3 corrects webhook scope and watch versions

A webhook configuration can describe a policy which the API server never actually applies. Kubernetes 1.9.3 fixes one such case for cluster-scoped resources. The February 9 maintenance release also includes a watch-cache correction that affects how clients resume observing changes. Both fixes matter to the idea of Kubernetes as an extensible control system. An extension needs the API server to invoke it for the right requests and report changes with the right versions.

I put considerable weight on Kubernetes API machinery because custom controllers depend on it. These fixes deserve attention even without a new workload type or a new configuration format. They correct behavior at the points where independently written code depends on the API server's interpretation of an object's scope and history.

A namespace selector was excluding cluster objects

The webhook fix in pull request 58185 changes a short branch in the namespace matcher. Before the change, a request with an empty namespace and a resource other than namespaces returned a non-match. That branch exempted cluster-scoped resources from webhooks during namespace matching. The corrected branch returns a match instead.

This does not mean every webhook now receives every cluster-scoped operation. Webhook rules still describe the operations and resources of interest. The correction removes an exemption in namespace matching which had prevented otherwise relevant webhook configurations from taking effect. A rule for nodes or PersistentVolumes can therefore start affecting requests after this patch.

Namespaces themselves have separate handling. Although a Namespace is a cluster-scoped object, the selector matches its own labels. Other cluster-scoped objects do not have a containing namespace whose labels the matcher can inspect. The patched API documentation says those objects do not skip the webhook through the namespace selector. That distinction prevents a namespace filter from silently deciding the coverage of unrelated cluster resources.

For namespaced objects, a selector can still express a condition on the containing namespace. This fragment illustrates the field described in the patched API documentation.

namespaceSelector:
  matchExpressions:
    - key: environment
      operator: In
      values:
        - prod
        - staging

The selector describes namespace membership for the matching decision. It should not be interpreted as an exclusion rule for every object which lacks an environment label. In particular, a node is not a namespaced object with a missing label. Its scope sends it through the corrected branch before namespace-label matching occurs.

That is why this bug fix has an upgrade consequence. An existing webhook may contain cluster-resource rules which have never been exercised by those requests. Correcting dispatch makes the existing configuration effective. Operators need to examine the rules and the webhook implementation together, since behavior previously hidden by the exemption becomes part of request processing. Deleting an unwanted rule and fixing a webhook that mishandles the resource are different responses to the same newly visible configuration.

A deletion event needs the version of the change

The watch-cache fix in pull request 58547 addresses a different extension boundary. Previously, the deleted event reused a deep copy of the previous object without replacing its resource version. The result carried the object's last update version, even though the event represented a subsequent change. The patch preserves the previous object's content while assigning the event's resource version to the returned object.

The distinction is easiest to see in the patch's filtering tests. A pod initially matches a watch filter at resource version 1. An update at version 2 changes the relevant field so it no longer matches. For that filtered watcher, the transition is a deleted event. The corrected expected event carries version 2, where the old expectation carried version 1.

A filtered deletion therefore need not mean that the underlying object has ceased to exist. It can mean that the object has left the set represented by the watch. The previous object's content is useful for identifying what left that set, while the event version identifies the change which caused it to leave. Using the older version confuses those two purposes.

The implementation supplies a versioner to each cache watcher. In the deleted-event branch it deep-copies the previous object, updates that copy with the event version and sends the copy as the watch event's object. This keeps cached previous content separate from the metadata sent to the client. The accompanying tests change their expected deletion versions and provide a test versioner which writes metadata.resourceVersion.

I find this correction especially relevant to the reconciliation model. A controller's response to an event may be to examine present state, but the watch stream still needs coherent continuation information. The fix makes the event describe the transition at which the watcher observed removal, including removal caused by filtering. It gives clients the change's version while retaining the object content needed to identify the affected entry in their own view.