Python 3.7.3 rejects ambiguous Unicode URL authorities

A URL parser can disagree with a later normalization step about which characters separate the host from the rest of a URL. Python 3.7.3 adds a rejection rule for that disagreement. The interesting part of this maintenance release is that a security correction changes what a familiar library function will accept, without adding a new API.

The Python 3.7.3 release is dated March 25, 2019. Its versioned release notes describe the change under bpo-36216. The correction makes urlsplit() raise ValueError when characters decompose through NFKC normalization into characters that affect URL parsing. The note associates that normalization with IDNA encoding.

This matters whenever software examines a URL before another component interprets it. A check on the apparent host is only useful if the later interpretation preserves the same boundary. Accepting an input and returning its components does not establish that later transformations will interpret those components identically.

I value Python's standard library for supplying implementations of protocols that applications should not need to reconstruct. That also makes a change like this relevant to the choice of interpreter patch version. The application can keep calling the same function while the runtime changes which inputs it considers acceptable.

The check happens in the authority component

The 3.7.3 parser implementation makes the mechanism concrete. After splitting the URL, urlsplit() passes its network-location component to _checknetloc(). The function immediately returns for an empty component or one containing only ASCII characters. Non-ASCII input receives the normalization check.

The implementation computes the NFKC form with unicodedata.normalize(). If normalization leaves the component unchanged, that check returns. Otherwise, it looks for characters that have structural meaning in a URL, including the slash and the characters used for query and fragment boundaries. A colon or at sign also causes rejection in the normalized value.

The source comment gives a particularly direct example. The Unicode character U+2100 expands to a/c under this normalization. A character that initially looks like part of a name can therefore introduce a slash. Splitting first and normalizing later can produce a different interpretation from normalizing before the split.

This is a validation check, not a rewrite of the supplied address. The parser does not substitute the normalized network location and continue as though the caller supplied it. It raises an exception for the rejected case. That behavior makes the uncertainty visible to the caller at the point where it asks for URL components.

An accepted call can become an exception

The following example illustrates the newly rejected case. It uses the character escape so the input remains visible in a source review.

from urllib.parse import urlsplit

try:
    parts = urlsplit("https://example.com\u2100archive")
except ValueError:
    print("The URL authority is not acceptable")

The example does not connect to a host. It exercises the parsing boundary described in the release notes and source. The exception is relevant to callers that previously assumed parsing always returned a result for a string of this form. Handling an invalid URL now includes this normalization-related rejection.

A successful parse still needs application policy. For example, obtaining a network location is separate from deciding whether an application should connect to that location. The patch addresses a specific disagreement about delimiters. It does not provide an application-specific list of destinations or decide which URL schemes a service should allow.

The distinction is useful when evaluating compatibility. A newly raised exception may reveal a previously unhandled input category. Suppressing it and continuing with the original string discards the reason the parser refused the input. Replacing the input with a different spelling is also a policy decision, because it can change the address the application ultimately uses.

The same release repairs other trust boundaries

The release notes contain a separate cookie correction under bpo-35121. With http.cookiejar.DefaultCookiePolicy, a cookie without a Domain attribute should not pass from one domain to another merely because their names satisfy the affected suffix relationship. This is another case where a library's interpretation of a name controls whether information crosses a boundary.

The URL and cookie corrections should remain separate in an upgrade explanation. One rejects a network location after inspecting Unicode normalization. The other changes cookie handling for a domain relationship. Their shared operational relevance is that routine library behavior can affect where data goes even when an application adds no new networking code.

The same notes identify a certificate-parser crash as CVE-2019-5010. Empty distribution-point or URI values in certificate revocation-list distribution points could cause a null-pointer dereference in the SSL module. That failure concerns processing certificate data, with a process crash as the documented consequence. It is not evidence that the URL parser correction changes TLS verification.

My preference is to explain a patch release through these specific contracts. Here, parsing some addresses becomes an error, cookie policy stops accepting an affected domain match, and certificate parsing handles a malformed case that could crash the process. Those are distinct changes that application and library maintainers can relate to their actual use of the standard library.

Python 3.7.3 consequently deserves attention even from applications that need no new language capability. The URL correction changes a concrete boundary between accepted input and an exception. Callers that process externally supplied addresses now receive an explicit failure for the affected Unicode authority forms.