Python 3.7.7 fixes duplicate thread identities in context caching

A crash in decimal arithmetic turned out to be a thread-identity problem. The Python 3.7.7 release includes a fix for duplicate thread-state identifiers created through PyGILState_Ensure(). Context-variable caching relied on those identifiers being unique. When the assumption failed, an apparently valid cache lookup could return an object belonging to another thread state.

The release's maintenance notes describe the race as observed under heavy thread load in embedded scenarios. That qualification matters. This is a problem with native threads entering an embedded interpreter through a particular API, and the report does not establish that every Python threading application experiences it. The affected code nevertheless sits beneath a useful standard-library abstraction.

I choose Python largely because its libraries provide capabilities that would otherwise require substantial implementation work. A fix like this deserves attention because application-level correctness can depend on interpreter bookkeeping far below the library call being made.

The original symptom pointed at decimal

Evgeny Boytsov opened issue 39776 on February 27 after crashes in a multithreaded application embedding Python. The initial reproduction used pybind11 to simplify the embedding. AddressSanitizer reported a use-after-free. Stefan Krah immediately identified an important version difference, explaining that the decimal threading code had moved to a ContextVar in Python 3.7.

The first discussion also shows why a crash location alone is weak evidence about its cause. GIL handling in the embedding example was an early suspect. A separate reproduction attempt produced an allocator error, and the participants had to distinguish that setup problem from the reported regression. Boytsov then removed pybind11 and the extra C++ module from his reproducer while retaining the failure.

By March 2, Krah had identified duplicate tstate->id values for thread states created through PyGILState_Ensure. The cache could treat two different thread states as the same owner. His later comment raised the possibility of incorrect results before a crash, because the test generated duplicate identifiers before it eventually failed. That is an important distinction between the observed crash and the wider correctness risk inferred from the implementation.

What the cache actually checks

The released context-variable implementation makes the dependency on identity explicit. PyContextVar_Get checks whether a variable has a cached value. It then compares the cached thread-state identifier with the current thread state's identifier and compares the cached context version with the current context version.

When both comparisons match, the function returns the cached value without searching the context's mapping. Otherwise it looks up the variable in the current context and records the result together with the identifier and version. The cached pointer is borrowed, as the comments in the source specify. The ownership and validity conditions therefore matter to memory safety as well as to which value the caller receives.

The version check cannot compensate for a duplicate identity. Two thread states can have the same context version while representing different contexts. In that case, an identifier collision can make the combined checks appear valid. This explanation follows from the cache comparisons in the released code and the race described in the issue.

A cache is only correct when the conditions used to recognise a reusable result distinguish every relevant state. Here the identifier was part of that condition, so allocating it safely was a requirement of context lookup. The arithmetic operation happened to expose the failure because decimal used the context machinery.

The repair protects allocation of the identifier

Krah's March 3 backport to the 3.7 branch is small. It moves the increment of interp->tstate_next_unique_id inside an existing HEAD_LOCK() region in new_threadstate. The same protected region already updates the interpreter's linked list of thread states.

The repaired order includes the following operations. This excerpt omits surrounding initialisation but preserves their order in the patch.

HEAD_LOCK();
tstate->id = ++interp->tstate_next_unique_id;
tstate->prev = NULL;
tstate->next = interp->tstate_head;

Previously the increment occurred before the lock. Moving it makes identifier allocation share the protection used when inserting the state into the interpreter's thread-state list. The patch does not change the cache's identity and version comparisons. It repairs the uniqueness assumption on which those comparisons depend.

I prefer this kind of explanation for a maintenance change. A precise invariant and the operation that restores it tell an embedding author more than a general claim that concurrency has improved. It also gives a reason why rewriting an application's decimal calculations would address the wrong component.

What the release evidence establishes

The issue records the 3.7 backport reaching the branch before the release-candidate cutoff. Boytsov subsequently reported that he could no longer reproduce the bug with either the test example or the production code after applying the patch. Those are reports from the participants, not a universal guarantee about unrelated embedding code.

For an application that embeds Python and creates native threads, the relevant maintenance change is below its Python modules. Both the issue and the released source identify interpreter thread-state identity as the mechanism. A context-variable cache needs unique thread-state identifiers even when all of its own comparisons are implemented as intended. Python 3.7.7 protects the increment that supplies those identifiers with the existing thread-state lock.