Go bounds HTTP/2 handlers after rapid stream resets

An HTTP/2 stream can finish from the protocol's perspective while the application handler started for it is still executing. Go's Rapid Reset vulnerability exploited that difference in lifetime. A client could create a request, reset its stream, and create another request while the first handler continued running. A stream-concurrency limit alone did not bound the accumulated application work.

The October 10 announcement releases Go 1.21.3 and Go 1.20.10 to address this problem. Go tracks it as CVE-2023-39325, with the wider issue also tracked as CVE-2023-44487. Users manually configuring HTTP/2 receive the corresponding fix in golang.org/x/net/http2 v0.17.0.

I value Go's concurrency model because the source makes concurrent operations relatively easy to identify. Resource limits still need to describe the lifetime of those operations accurately. A goroutine started for a request remains server work even after the client has stopped treating the stream as active.

Stream limits and handler limits count different objects

The Go issue description identifies the mismatch directly. MaxConcurrentStreams bounded streams on a connection. Resetting an in-progress stream allowed the attacker to create another stream, while the existing request handler could still be executing.

There is no contradiction in those observations. A stream belongs to the HTTP/2 connection state. A handler is application execution started on behalf of that stream. The client can change the former state faster than the application necessarily completes the latter. Accounting that releases capacity when the stream resets can therefore admit work before the earlier work has finished.

Consider a server that starts a handler for each accepted request. If every reset immediately permits another request to start a handler, the number of active handlers can grow even while the number of live streams stays within the configured limit. The stream count remains accurate about streams. It is simply insufficient as a bound on execution.

That distinction explains why a general statement that the server has a concurrency limit is incomplete. The object being counted and the event that releases its slot determine what the limit actually protects. The disclosed problem concerns handler work that outlives the protocol object used for the original accounting.

The fix reserves capacity until a handler exits

The corrected server bounds simultaneously executing handler goroutines using the stream-concurrency limit. A request arriving while that handler limit is reached waits in a queue. The announcement notes that reaching this condition requires an earlier in-flight request to have been reset, because otherwise the stream limit already prevents the extra request.

The v0.17.0 server implementation makes the accounting explicit in scheduleHandler. If the number of current handlers is below the advertised stream maximum, it increments that number and starts a handler goroutine. Otherwise it records an unstarted handler for later consideration.

The handler count falls when a handler completes. runHandler defers a completion message to the connection's serving logic. That message leads to handlerDone, which decrements the count and considers queued requests. The counter therefore follows completion of execution, rather than the client's decision to reset a stream.

This is the central correction. A reset cannot by itself reclaim the slot occupied by a handler that has not returned. A new stream may exist, but starting more application execution requires handler capacity. The server can distinguish protocol progress from completion of the work initiated by that protocol.

The implementation also handles a queued stream that is reset before its handler starts. When draining the queue, it checks whether the stream still exists. If the stream has disappeared, the server skips that queued handler. Work that has never begun need not start merely because it once entered the queue.

That check matters under the same pattern of rapid cancellation. Without it, a queue could preserve obsolete work and start it after the corresponding request had ceased to matter. The fixed code uses the current stream state when deciding which waiting handlers should consume newly available execution capacity.

A queue needs its own resource bound

Limiting executing goroutines would be incomplete if the waiting queue could grow without restriction. The server would then exchange one accumulation of resources for another. The fix consequently terminates a connection when its queued work grows too large.

In the versioned implementation, scheduleHandler tests the queue length against four times the advertised maximum stream count before appending another unstarted handler. Exceeding that check produces a connection error. The announcement describes the behavior at the operational level as terminating a connection with an excessive request queue.

The exact comparison is an implementation detail, so it should not be mistaken for a new application-facing promise about a precise allowable backlog. Its purpose is clear in the code. Waiting requests consume resources too, and an abusive connection cannot retain an unlimited number of them simply because the active-handler count is bounded.

The two controls protect different states of work. Counting handlers limits execution already started. The queue check limits requests admitted to wait for execution. Treating both states explicitly prevents a client from using rapid resets to move resource consumption into a category that the server does not count.

I find this more useful than describing the fix as a generic request-rate limit. It does not require every normal connection to obey one requests-per-second number. It bounds active and waiting work whose lifetimes the server can observe, which is the relationship the attack violated.

Queue maintenance also clears references in entries whose handlers have started, and resets the queue slice when it becomes empty. The implementation therefore manages both admission and retention of waiting-request data. Completing the transition out of the queue includes giving up references that the queue no longer needs.

The default is a per-connection bound

The announcement identifies a default limit of 250 concurrent streams per HTTP/2 connection. It also points to http2.Server.MaxConcurrentStreams and ConfigureServer for users who need to adjust it through golang.org/x/net/http2. The correction uses that existing concurrency setting for handler accounting.

The words per connection matter. A bound enforced on one connection does not establish a process-wide handler budget across all connections. The disclosed repair prevents the reset pattern from evading the intended limit within a connection. It does not claim to supply every admission-control policy an application or deployment might need.

The configured value also has a concrete relationship to application behavior. Handlers that remain active longer occupy their slots longer, including after a client reset. Under the correction, that fact becomes visible as waiting work and eventually a connection-level rejection if the backlog grows excessively. Capacity no longer appears to recover merely because the client discards the protocol stream.

This behavior preserves an important distinction for operations. A slow or blocked handler is still consuming a place in the server's execution budget. The accounting remains tied to actual completion. It does not infer that the work has ended from the client announcing that it no longer wants the response.

Applying the correction depends on the HTTP/2 implementation in use

The release offers fixes in both maintained Go release families and in the separately versioned HTTP/2 package. For a server using the standard library's implementation, the patched Go toolchain supplies the corrected library code to a subsequent build. A server manually using x/net/http2 needs the fixed module version identified by the announcement.

Those paths are related but need separate version checks. Knowing that an application uses Go does not identify every HTTP/2 implementation included in it. The announced x/net release exists because users can configure that package directly, outside the standard library copy.

My preference is to describe the upgrade in terms of the actual implementation compiled into the server. That connects the advisory to the artifact that will accept connections. The relevant corrected behavior is the separate handler count and bounded waiting queue, regardless of which supported distribution path supplied that code.

The fix also provides a useful way to read concurrency controls elsewhere in a system. A client-visible operation and the computation it started can end at different times. Here the code now releases execution capacity when the handler reports completion, while independently checking whether waiting streams still exist. Those are the state transitions that keep rapid protocol cancellation from creating unbounded handler execution.