Python 3.8 beta exposes shared memory ownership

Sending a large value to another process can involve work that has little to do with the computation itself. Python 3.8's new shared-memory module provides a way for processes to access the same bytes. The useful distinction is between transferring a representation of data and arranging access to a common allocation.

Python 3.8.0b1 arrived on June 4, 2019 as the first planned beta. Its announcement asks third-party maintainers to exercise the developing release and identifies shared memory as a way to avoid pickling costs between processes. The beta is a preview, so the available API is an implementation to investigate rather than a production release.

The module documentation at the beta tag explains the mechanism. A process creates a named block through SharedMemory. Another process attaches using the same name. Each instance exposes the contents as a memoryview through buf. The name supplies an attachment mechanism, while the buffer supplies access to the data.

I find this relevant to why Python remains useful despite its surrounding complexity. Libraries often provide the difficult implementation, particularly around numerical work and native code. An explicit buffer interface gives those libraries a way to use common storage without requiring each one to invent its own naming and allocation API.

A name identifies an allocation

Creating a block uses create=True and a requested size in bytes. Omitting a name asks the module to generate one. Attaching uses an existing name with the default create=False. The documentation says the size argument is ignored when attaching, because the block already exists.

The requested allocation can be smaller than the actual block if the platform allocates in page-sized chunks. The API exposes the resulting size separately. Code therefore needs to distinguish its logical data length from the capacity of the allocation. Extra allocated bytes do not automatically become meaningful elements of an application record.

A small same-process example demonstrates attachment without introducing a process-launching framework.

from multiprocessing import shared_memory

owner = shared_memory.SharedMemory(create=True, size=5)
reader = None
try:
    owner.buf[:5] = b"hello"
    reader = shared_memory.SharedMemory(name=owner.name)
    print(bytes(reader.buf[:5]))
finally:
    if reader is not None:
        reader.close()
    owner.close()
    owner.unlink()

Both instances refer to the same block. Converting the view to bytes makes a copy for the printed value. The example is intentionally limited to demonstrating the storage API. The module's documentation also shows attachment from separate interactive processes using the generated name.

Shared bytes need an agreed interpretation

The buffer does not carry an application schema. A numerical array needs a shape and an element type as well as the storage containing its values. The beta documentation's NumPy example supplies those properties when constructing an array backed by the shared buffer. A second process reconstructs a compatible view using the same shape and type.

That arrangement avoids repeatedly serializing the array contents for transfer. It still requires agreement about what the bytes mean. Supplying a name alone does not tell another program whether the allocation contains integers, text or a structured record. Metadata can remain part of a separate coordination mechanism even when the large payload stays in shared memory.

The example also copies an existing array into the allocation before another view accesses it. Shared memory does not retroactively make every existing Python object shared. The storage must actually reside in the common block, and a library needs a supported way to use that storage.

This distinction limits broad performance claims. Avoiding a serialized payload can remove particular copies and encoding work. It does not establish that every computation will run faster, and the module documentation does not supply a universal speedup. Its mechanism is concrete enough to explain without inventing a benchmark.

Closing access differs from destroying storage

The API makes lifetime explicit. close() releases access through one instance without destroying the block. unlink() requests destruction of the underlying allocation. The documentation requires each instance to close when it is finished and calls for unlinking once across the processes using the block.

Those are different ownership responsibilities. A reader can finish before the rest of the application has finished with the data. Releasing that reader's access must not imply that all other readers have finished. Conversely, closing every local object without arranging destruction does not express the complete cleanup operation described by the API.

The documentation notes that the block can outlive its creating process and that destruction timing varies across platforms. Access after unlinking can cause memory access errors. These conditions mean the application needs a defined point at which all participants are finished with the allocation.

I prefer the API's explicit separation because it exposes the resource model. A handle has a local lifetime, while the allocation has a shared lifetime. Combining them would make one process's cleanup decision harder to reason about when other processes still require the data.

A manager can own the cleanup policy

The beta also documents SharedMemoryManager in multiprocessing.managers. Starting it creates a process whose role is managing shared-memory lifetimes. Shutting it down calls unlink on the blocks it manages and stops that process. A context-manager form can tie this ownership to a lexical scope.

This provides a useful alternative to manually tracking every allocation. It does not remove the need to finish work before cleanup. The documentation's example joins worker processes before leaving the manager's context, so the lifetime of their work fits inside the lifetime of the storage.

The first beta makes this model available for examination alongside the rest of the developing interpreter. Processes can share named bytes while retaining responsibility for their meaning and the point when storage is no longer needed. Library authors now have a standard allocation and attachment API around which to define those higher-level agreements.