Kubernetes 1.12.3 checks the backend before proxying an upgraded connection

A request to upgrade a connection does not establish that the backend agreed to the upgrade. The distinction appears directly in the upgrade-aware proxy shipped with Kubernetes 1.12.3 on November 26. Before it starts copying data in both directions, the handler now reads the backend's HTTP response and requires the switching-protocols status. This post examines that behavior from the released implementation.

I care about this distinction because Kubernetes delegates work across process boundaries throughout the system. The state requested by one component and the result acknowledged by another are separate facts. A proxy participates in that coordination even when its eventual job is to copy bytes. It needs to establish which kind of connection it has before treating the connection as an upgraded stream.

An upgrade begins as an HTTP exchange

The version-pinned proxy source begins its upgrade path by checking whether the incoming request asks for an upgrade. When it does not, the method returns without handling the request through this path. An upgrade request then acquires a backend connection through the configured location and transport behavior.

The handler can use the incoming request location while preserving the configured backend scheme and host. It clones the request and appends forwarding information for the upgrade path. Depending on configuration, it either connects with redirect handling or dials directly. That means the response-reading stage may receive both a connection and bytes already read while establishing it.

The code handles that possibility with io.MultiReader. It combines any previously captured response bytes with the remaining backend connection and passes the result to a helper which parses an HTTP response. This avoids assuming that the entire response header is still waiting untouched on the socket. The relevant input is the logical response stream, including bytes consumed by earlier connection setup.

The helper returns the status code together with the raw bytes consumed during parsing. If parsing fails, the handler reports the error and returns before taking over the client connection. A backend which does not supply a readable HTTP response cannot reach the normal upgraded-stream path through this sequence.

Reading headers must preserve the bytes being forwarded

The response helper uses a buffered reader over an io.TeeReader. As HTTP parsing consumes data, the tee records the same bytes in a buffer. The code can then inspect the response status while preserving the original response bytes for the client. This is necessary because parsing changes the read position on the backend stream.

A buffered parser may read more bytes than the minimal header representation. The handler compares the number of captured bytes with the response bytes it already held and expands its saved response when necessary. Returning only the status code would leave consumed bytes unavailable to the later forwarding path. Capturing them keeps inspection from silently deleting part of the backend's response.

This gives the helper two responsibilities which belong together. It establishes whether the backend accepted the protocol transition, and it accounts for the bytes read while making that decision. A proxy cannot safely treat response inspection as a side-effect-free operation on a stream. It has to preserve what the downstream client still needs to receive.

The source's central decision is small.

if rawResponseCode != http.StatusSwitchingProtocols {
    // Return the backend response without starting bidirectional copying.
}

This excerpt illustrates the decision in the released handler rather than providing a standalone proxy implementation. The surrounding code writes the saved response bytes and handles the rest of the backend response. The condition matters because a successful network connection alone says nothing about whether the HTTP upgrade succeeded.

Connection takeover happens after response inspection

The handler checks whether the response writer implements HTTP connection hijacking. It then takes over the client connection, with deferred cleanup for both client and backend connections. A source comment explains that the normal error responder no longer works after hijacking, so this action belongs late in the upgrade sequence.

That ordering keeps parse and connection-establishment failures within the ordinary response mechanism. Once the handler owns the raw client connection, it has to report and forward results through that connection itself. Reading the backend status before takeover reduces the work performed after crossing that boundary, while the captured response bytes permit forwarding to continue.

After takeover, the handler writes any saved raw response bytes to the client. For a backend status other than switching protocols, it copies the remaining backend response toward the client and returns. It does not start the client-to-backend copying goroutine in that branch. Returning also invokes the deferred connection cleanup.

For a switching-protocols response, the implementation starts copying in both directions. It uses separate completion channels for each direction and returns when either completes. Deferred cleanup closes the other side as the method exits. The code also supports optional flow-rate wrappers when a maximum byte rate is configured.

These details describe a specific state transition. The client asks for an upgrade, the backend supplies its result, and only an accepted upgrade reaches the bidirectional stream copy. A rejected upgrade can still return its response to the client. It simply does not acquire the same forwarding behavior as an accepted transition.

I find the byte accounting as relevant as the status check. A correct decision is insufficient if the inspection consumes data the client never receives. The implementation records the bytes read during parsing and writes them before processing the remaining connection. Protocol state and stream position have to remain consistent throughout that handoff.

CSI capacity needs the right API update path

Another correction in this release concerns how storage information becomes visible to the control plane. The CSI attachment-limit fix changes node information updates from a general Node update to a node-status patch. Its source comment states the problem directly, updating status through a normal Node update does not work.

The patch reads the original Node and creates a deep copy for changes. When changes are needed, it calls PatchNodeStatus with both the original and modified object. This gives the helper the before and after representations needed to construct the update. The comment also notes that this helper can update labels or annotations along with status.

This is a different instance of a requested change needing the correct mechanism to become actual state. Computing an attachment limit locally does not establish that the API's node capacity and allocatable information now contain it. The writer has to use the update path which accepts those status changes. Otherwise, a driver can report information which never reaches the representation other components inspect.

The accompanying test changes are informative. They inspect client actions for a patch and apply that patch to the original Node before checking the result. That makes the test evaluate what the status patch represents. Merely retrieving a modified object from a fake client would not establish the same behavior at the API operation boundary.

The test helper serializes the original Node, applies the strategic merge patch and decodes the result back into a Node. It then validates the resulting information using the existing cases. This focuses the test on the form of mutation sent by the implementation. It also preserves a separate path for cases where no patch action occurs.

The proxy and CSI corrections affect different parts of a cluster, but both require an explicit account of a component boundary. Backend response inspection precedes stream forwarding. Node information updates use a status patch before attachment limits become recorded node state. In each implementation, the operation chosen at that boundary determines whether the locally intended change is represented correctly to the next component.

Azure disk operations remove an internal retry layer

The Azure disk patch included in this release removes explicit backoff retry calls around virtual-machine updates during attachment and detachment. It makes corresponding changes for ordinary virtual machines and scale-set instances. Successful operations still invalidate the relevant machine cache, so subsequent reads do not rely on the pre-update representation.

The patch also includes disk names in several operation log messages. That makes the messages describe which disk the update concerns along with the machine or node. Failed attachment handling still checks for lease and missing-blob errors and attempts detachment in those cases. Removing the retry block therefore leaves other operation-specific error handling visible in the same functions.