Go 1.16.4 removes recursion from HTTP header token scanning
CVE-2021-31525 lets a very large HTTP header cause an unrecoverable panic. The repair changes token scanning, with different exposure for clients and default server configurations.
An HTTP client accepts data from a server whose behavior it may not control. That trust boundary matters in Go 1.16.4's security fix. A sufficiently large response header can drive recursive token scanning into an unrecoverable panic. The default header limit protects an ordinary Go server from the corresponding request path, but it does not make a Go HTTP client immune to a malicious server.
The May 6 release announcement identifies the issue as CVE-2021-31525 and Go issue 45710. It announces Go 1.16.4 and Go 1.15.12. The release history records the net/http security fix alongside maintenance changes in the compiler and other packages.
I like Go's standard HTTP implementation partly because it makes useful networking code accessible without a large supporting framework. That also makes the distinction between client and server exposure important. Familiar package names do not imply that both directions apply the same limits.
Recursion follows the number of tokens
The release diff shows the relevant change in the vendored httpguts implementation. Its headerValueContainsToken helper looks for a token among comma-separated values and compares them without regard to ASCII case. The old implementation finds a comma, compares the preceding portion, and calls itself with the remainder when necessary.
That algorithm makes nesting depth depend on the number of comma-separated portions it must examine. The code processes less input at each call, so it has an eventual stopping condition. Termination alone is insufficient, however. Each recursive call requires execution state before the eventual return can unwind the sequence.
The repair performs the same scan with a loop. A matching token returns true immediately. Otherwise, the working string advances past the comma and the next iteration examines its remainder. After no commas remain, the final portion receives the same comparison. The scanner retains its whitespace trimming and case-insensitive matching behavior.
The change is significant because repeated input no longer requires repeated function nesting. An iterative scan still has work proportional to what it examines. It removes the call-stack growth caused by expressing that work recursively. This addresses the mechanism in the patch without claiming that arbitrarily large headers become free to process.
The patch is also small enough to compare semantically. Both forms stop early when they find the requested token. Both compare the final comma-free portion. Replacing recursion here does not require a new representation of headers or a change to the calling API. It changes the resources used to traverse the same input.
The advisory names the affected entry points
ReadRequest and ReadResponse can encounter the panic while reading a very large header. The announcement gives approximate trigger sizes above 7 MB on 64-bit architectures and above 4 MB on 32-bit architectures. These figures describe the reported failure conditions. They are not recommended maximum header sizes for an application.
Transport and Client are vulnerable, according to the advisory. A malicious server can therefore make the client program crash. This is an important part of the impact statement because an application may use HTTP only for outbound requests. Restricting attention to public listeners would miss that use of the affected parser.
A normal server has a different default. The announcement says Server is not vulnerable by default because its maximum header size is 1 MB. Raising Server.MaxHeaderBytes sufficiently can expose it to a malicious client. Configuration therefore changes the reachability of the vulnerable behavior even though both configurations use the same package implementation.
This does not justify treating the default as a universal protection. Direct callers of ReadRequest or ReadResponse need to understand their own input boundary. Client users need the fixed code. A server whose limits have been enlarged needs to account for that decision. The relevant question is which parser receives data and what constrains it before the recursive scan occurs.
The module copy needs attention too
The announcement explicitly includes golang.org/x/net/http2/h2c and HeaderValuesContainsToken in golang.org/x/net/http/httpguts. It identifies a fixed x/net revision with the pseudo-version v0.0.0-20210428140749-89ef3d95e781. That means the security notice concerns a dependency as well as the standard library distribution.
The Go 1.16.4 diff updates the toolchain's vendored x/net code and its module metadata. An application with its own x/net dependency has a separate dependency selection. Rebuilding with a repaired standard library does not by itself describe which x/net revision the application's module selects. The two copies need to be distinguished when determining which implementation reaches the executable.
This is a useful example of why a release number and a module version answer different questions. The toolchain release identifies the standard library implementation used for a build. A module requirement identifies another source of package code. An exposure assessment needs the code path and the version of the implementation on that path.
I prefer fixes that preserve a clear reason for the behavior change. Here the reason is inspectable in a short helper. Repetition in an untrusted header used to produce nested calls. The repaired implementation consumes successive portions in one function invocation, while preserving its token comparison.
Limits and parser repairs solve different problems
The server default demonstrates the value of bounding input before expensive processing. The parser repair demonstrates why that bound cannot replace implementation correctness across all callers. A limit belongs to the context using the parser. The helper still needs a reasonable algorithm when a caller legitimately supplies a larger value.
The announcement credits Guido Vranken with reporting the crash through the Ethereum 2.0 bounty program. Its impact description is an unrecoverable panic, so ordinary application error handling is not the repair. The affected code needs the updated implementation, and builds using x/net need the appropriate dependency revision as well.
For an application that communicates with external HTTP endpoints, the substantive consequence is that response handling is part of its availability boundary. An outbound connection does not establish that the returned headers are safe to process. Go 1.16.4 changes the scanning mechanism that made this particular input shape capable of crashing the program.