Python 3.8.2 restores pickle support for minimal file objects

A file-like object does not necessarily implement every operation that a built-in file supports. Python's libraries have to decide which methods an operation actually requires. Python 3.8.2, released today, repairs a regression at that boundary in the C implementation of pickle. Objects without readinto() can once again supply input for unpickling. The failure reached beyond code calling pickle directly, through the serialization routines of other libraries.

The interesting part is how a reasonable implementation decision became an incompatible public requirement. In issue 39681, Nathan Goldbaum reported that the C implementation had started expecting readinto. Antoine Pitrou explained that the PEP 574 implementation review had considered requiring a proper file implementation, for example one derived from io.BufferedIOBase. Existing callers had relied on a smaller protocol.

I like Python's ability to let a small object satisfy a library protocol. It makes useful adapters possible without forcing an inheritance hierarchy onto every caller. That benefit depends on the library keeping its actual requirements stable, including the requirements inside its accelerated implementation.

How the regression reached application code

Goldbaum's February 19 report described failing PyTorch tests that explicitly covered a file-like object without readinto. The issue was therefore more specific than an accidental omission in an application wrapper. A downstream library had tests for the narrower interface and expected that behaviour to remain usable.

Further investigation in the same thread traced an affected path through flair and PyTorch to a memory-mapped file. Goldbaum identified the absence of readinto on the mapping as the immediate cause. Passing that object through several layers did not change its interface. The new requirement only became visible when the C unpickler inspected it.

This is a practical consequence of protocol-based APIs. The object that reaches the standard library may have been supplied by an application several calls earlier. A new method requirement can therefore break a higher-level operation whose caller never knowingly selected a particular pickle implementation. The traceback describes the eventual failure, while the compatibility decision belongs to the library defining the protocol.

The 3.8.2 changelog identifies this as a regression and records its repair in the final release. That classification matters. Existing minimal objects regain support through a maintenance release, without requiring every downstream caller to adopt a larger interface.

Optional capabilities in the implementation

The released C unpickler implementation makes the distinction visible in _Unpickler_SetInputStream. It looks up peek and readinto as optional methods. It also obtains read and readline, then rejects an input object if either of those required attributes is missing.

An optional method lets an implementation use a capability when an object supplies it. Requiring that same method during initialisation imposes a broader contract on every caller, even callers whose input could be consumed through the existing operations. The fix restores that distinction in the C implementation.

A deliberately small adapter illustrates the supported case. This example serializes a locally created object and exposes only the stream methods that the fixed initialisation requires. It also asserts that the adapter has no readinto attribute, so the example cannot pass merely because a base class supplied one.

import io
import pickle

class MinimalReader:
    def __init__(self, data):
        self._stream = io.BytesIO(data)

    def read(self, size=-1):
        return self._stream.read(size)

    def readline(self, size=-1):
        return self._stream.readline(size)

expected = {"workers": 4}
reader = MinimalReader(pickle.dumps(expected, protocol=4))
assert not hasattr(reader, "readinto")
assert pickle.load(reader) == expected

The wrapper uses composition to make its supported operations explicit. Passing BytesIO directly would not demonstrate this regression because that object supplies a richer interface. A regression example needs to preserve the absence of the optional method as well as the successful result.

A memory mapping has another interface available

Restoring compatibility does not establish that every compatible usage is the best way to consume a particular object. In the issue discussion, Pitrou suggested pickle.loads(memoryview(mm)) for the memory-mapped case. His explanation was that this reads from memory directly instead of making read() calls on the mapping.

That distinction is specific to an object that can provide memory through the buffer interface. An arbitrary stream wrapper cannot adopt the same expression simply because it supports read. Stream consumption and access to an existing memory buffer offer different capabilities. The appropriate operation follows the capabilities of the input object.

I appreciate that the repair preserves the old protocol while the discussion still identifies a better operation for the mapping case. Compatibility and an API recommendation answer separate questions. A maintenance release can restore working callers while library authors decide how to express a more suitable data path.

Python's ecosystem is a major reason to choose it, so a standard-library protocol change has consequences across dependencies. This repair is useful precisely because it addresses the shared implementation. A PyTorch caller using a minimal reader and a library author revising a memory-mapping path can both make progress without requiring the same interface change.

The resulting contract is visible in the released source. The C unpickler requires read and readline during input setup and treats readinto as optional. Adapters that intentionally expose the smaller protocol can continue to work with Python 3.8.2.