Python 3.10.1 repairs escape decoding across chunks

An input chunk can end halfway through a valid escape sequence. Treating that boundary as the end of the encoded text makes decoding depend on how a reader divided its input. Python 3.10.1, released on December 6, repairs that behavior in the unicode-escape and raw-unicode-escape codecs. Both incremental decoders and stream readers receive fixes for split sequences.

The maintenance changelog records the changes under issues 45461 and 45467. Their importance lies in the codec contract. A stream reader cannot generally choose a byte boundary that also happens to be a complete encoded-character boundary. The decoder must retain enough state to complete a sequence when more input arrives.

I appreciate this part of Python's protocol design because it lets a reader work in bounded pieces without requiring every caller to understand the encoding's grammar. That separation only holds when incomplete input remains distinguishable from invalid input. Otherwise, a correct document can fail for reasons determined by buffering.

Decoding has a state between calls

The codec documentation shipped with this release describes IncrementalDecoder.decode as taking both new input and a final flag. The decoder considers its existing state when processing the new bytes. Calls normally leave final false, while the last call must mark the input complete.

That flag changes what an incomplete sequence means. Before final input, an unfinished escape can be a prefix of valid data. At final input, the decoder must finish processing and flush its buffers. If completion is impossible, it invokes its error-handling policy just as a stateless decoder would.

The release's codec tests exercise partial decoding for both affected codecs. They check repeated intermediate results while bytes arrive, including steps where no additional character can yet be returned. That is expected behavior when the decoder has received only part of the next character's representation.

A transport boundary does not complete an escape

Consider the bytes spelling a Unicode escape for the euro sign. The first call below ends before all hexadecimal digits arrive. The decoder has no complete character to return yet.

import codecs

decoder = codecs.getincrementaldecoder("unicode-escape")()
first = decoder.decode(b"\\u20", final=False)
second = decoder.decode(b"ac", final=True)

assert first == ""
assert second == "\u20ac"

The corresponding complete input is b"\\u20ac". Incremental decoding must preserve the meaning of that encoded text across the split. Returning an empty string from the first call does not mean the input was discarded. It means those bytes remain part of the decoder's pending work.

By contrast, declaring b"\\u20" final leaves the escape incomplete. With strict error handling, that condition requires an exception. Suppressing the error indiscriminately would confuse valid split input with permanently truncated input. The fix needs to preserve both outcomes, depending on whether more bytes may still arrive.

The same issue applies to stream readers. The codec documentation describes StreamReader.read in terms of requested decoded output and an approximate bound on encoded input. Those quantities are not interchangeable because an encoded sequence can require several bytes before it produces a character. A stream implementation therefore needs codec state even when its caller only sees ordinary reads.

Buffering is part of the codec interface

The incremental decoder's documented getstate returns a pair containing still-undecoded input and additional state. setstate can restore a state returned by that interface. These methods make pending bytes an explicit part of the decoder's behavior. They are not incidental leftovers that a caller can safely forget between chunks.

A consumer that creates a fresh decoder for every chunk defeats that stateful interface. Each instance sees its chunk as an independent input and cannot use bytes retained by the previous instance. The repaired codec still requires the caller to maintain one decoding operation across the chunks that belong to it.

This also explains why changing error handling is not an equivalent repair. Ignoring incomplete escapes can remove data, while replacing them can change the decoded result. Neither establishes the required correspondence between a valid complete byte string and the same string supplied incrementally.

I prefer tests that compare those two forms of input directly. The release's partial-decoding cases examine intermediate output as well as the completed string. That checks the decoder's state transitions, including the periods when it must retain bytes without producing a character. Moving the split to another byte position tests the same encoded value under a different delivery pattern.

Python 3.10.1 restores the intended handling of split escape sequences in these codecs. An intermediate chunk can leave a pending escape, and final input still requires that escape to become complete or enter error handling. Applications using the incremental or stream-reader interfaces can retain bounded reads without assigning semantic meaning to the accidental position of each read boundary.