Python 3.8.5 rejects zero-length PAX records in tarfile

An archive parser can hang without receiving a large archive. A record that claims to have zero length is enough if the parser uses that length to advance through its input. Python 3.8.5 repairs this exact failure in tarfile. The release is out of schedule because of its security content, and the archive fix is one of the changes it carries.

The release notes identify the infinite loop as CVE-2019-20907. Reading the underlying issue shows how the malformed input exploits a local parser invariant. A finite input buffer can leave the loop processing it without progress.

I choose Python for the libraries that already implement difficult formats and protocols. That makes their rejection behaviour part of the value I need from them. A parser has to define what happens when a field is syntactically recognisable but cannot describe a valid record.

The length includes the length field itself

The comments in the released PAX parser describe a record as a decimal length, a space, a keyword and value separated by an equals sign, and a newline. The declared length covers the complete record, including the characters that spell out the length. Both the keyword and value use UTF-8 encoding under the normal format rules described there.

The parser first matches the beginning of a record with a regular expression. It extracts the decimal length and keyword, converts the length to an integer, and uses the declared boundary to slice out the value. After processing the field, it advances pos by that length to locate the next record.

Those operations assume that a successful record consumes some input. The old pattern could recognise a zero in the length position. If the subsequent code accepted it, adding the parsed length to pos left the cursor unchanged. The next loop iteration then matched the same bytes again.

The important distinction is between recognising the textual shape of a field and validating its meaning. A sequence of decimal digits is suitable input to integer conversion. The resulting integer still has to satisfy the format and the parser's progress requirement. Matching the prefix alone cannot establish that.

A small reproduction isolated the failure

In issue 39017, the initial report describes finding the loop through fuzzing. A subsequent comment reduced the input to a normal PAX header block followed by a record prefix beginning with a zero length. The explanation identifies pos += length as the operation that fails to move the cursor.

That reproduction gives a more specific account than describing the problem as excessive archive processing. The loop repeatedly processes the same location. A small input-size limit would not necessarily address it because the issue does not require a large expanded file or a long sequence of records.

The issue also discusses a separate malformed case involving a very large declared length and an unexpected exception. That observation should not be conflated with the zero-length fix. The released change shown here establishes a guard against the non-advancing case. It is not a claim that every invalid PAX length now has identical error handling.

For a program that accepts archives, metadata parsing can therefore consume time before useful member data is available. The reported reproduction reaches the failure through tarfile.open. Avoiding extraction would not by itself avoid a loop encountered while opening and parsing that archive.

The fix stops before consuming a value

Python 3.8.5 adds the guard immediately after converting the length to an integer. The relevant excerpt is short and retains its actual exception type.

length, keyword = match.groups()
length = int(length)
if length == 0:
    raise InvalidHeaderError("invalid header")

The parser now rejects that header before slicing the value or advancing to the next record. This restores progress for the particular malformed case without imposing an arbitrary iteration count. The condition follows from the record's own declared length and the operation that uses it.

InvalidHeaderError is the exception at this internal parsing location. It should not be presented as a promise that every public tarfile operation exposes that exact exception unchanged. An API can translate internal parse failures while still relying on this guard to stop the loop.

I prefer the direct check here because its reason is visible beside the integer conversion. A later timeout could bound the consequences of a hang, but it would leave the parser accepting a record that cannot advance its own cursor. The implementation now rejects that case where the necessary information is available.

The release is more specific than a general security label

The release page asks users to upgrade because of important security content. For this archive issue, the practical change is identifiable without guessing at an application's exposure. PAX parsing no longer accepts a matched record length of zero and returns to the same input position forever.

The issue's denial-of-service description follows from that failure to terminate. It does not establish code execution or every other consequence that a security label might suggest. Applications that process supplied archives gain a parser correction at the shared library level, including when another package calls tarfile on their behalf. The guard executes during metadata processing, before the parser can repeat the same zero-length record.