Python 3.6 extends protocols into formatting and asynchronous iteration

Python's generators let a function express a sequence without manually maintaining an iterator object's state. Asynchronous iteration has needed more ceremony. A producer that waits between values has had to implement the asynchronous iterator methods itself. Python 3.6 extends the generator model to that case, so a function can await work and yield values in the same body. The change exposes a harder question about how suspended functions finish when their consumers stop early.

The Python 3.6.0 release page gives December 23, 2016 as the release date. Its new language facilities include asynchronous generators and formatted string literals. Both shorten familiar operations by using existing protocols. F-strings use the formatting behavior provided by objects. Asynchronous generators implement asynchronous iteration and require a scheduler to complete operations that can suspend.

I find Python most convincing when its protocols let library code participate in ordinary language constructs. These additions continue that approach. They deserve examination beyond their shorter spelling, because formatting can execute expressions and asynchronous cleanup can itself require asynchronous execution.

Formatting uses an object's existing behavior

PEP 498 explains why another string-formatting mechanism was proposed. Percent formatting has a trap when a single argument becomes a tuple, while str.format() can separate a replacement field from the expression providing its value. Named fields can require repeating the same name in a keyword argument. F-strings put an expression next to the surrounding text while retaining the format specification machinery.

count = 12
duration = 0.375
message = f"Processed {count} items in {duration:.3f} seconds"

The expressions inside braces evaluate when the statement runs. An f-string is therefore an expression that produces a string, even though its spelling resembles a literal. It does not create a reusable template that will evaluate its fields later. Changing count after assigning message does not update the string already produced.

The format specification after the colon goes through the object's __format__ protocol. This is the same mechanism used by str.format(). A type that already supports a useful formatting convention can keep that behavior when its callers adopt f-strings. The new syntax does not require every formatter to provide a second implementation.

This matters for objects whose formatted representation carries more structure than a plain conversion to text. PEP 498 uses dates and decimal values to explain the existing protocol's usefulness. A field can also request a conversion such as !r before formatting. That conversion changes which object receives the subsequent formatting operation, so it is part of the expression's meaning rather than decoration around the output.

Full expressions are allowed in replacement fields, including function calls. The proposal specifies evaluation in the scope containing the f-string and evaluates expressions in order. A string construction can consequently perform work or raise an exception while producing a field. The syntax has the same responsibility to make such execution understandable as any other expression in the function.

F-strings do not obtain the caller's complete local or global mapping and hand it to a formatting routine. PEP 498 rejects that approach and describes its problems with closures and excess access to scope information. Explicit expressions allow the compiler to identify the names that the code references. The shorter source still has identifiable dependencies.

There are also grammar restrictions to account for in this release. Backslashes and comments cannot appear inside replacement expressions. Literal braces need doubling. An expression that accesses a dictionary using a quoted key needs quoting that does not prematurely terminate the surrounding string. These are rules of the Python 3.6 syntax, and examples written for this release need to respect them.

A generator can wait before producing another value

PEP 525 describes the manual asynchronous iterator that generators can replace. Such a class maintains its iteration state and implements __aiter__ and __anext__. The latter returns an awaitable and signals exhaustion with StopAsyncIteration. Python 3.6 can generate that behavior for a function containing both async def and yield.

import asyncio

async def samples(delay, count):
    for index in range(count):
        await asyncio.sleep(delay)
        yield index

async def display():
    async for index in samples(0.1, 3):
        print(index)

Calling samples creates an asynchronous generator object. The async for loop drives its asynchronous iteration protocol. Execution can suspend at the sleep operation and can suspend again when it yields a value to the consumer. The function's local state remains available when iteration resumes.

The example defines the producer and consumer without starting an event loop. PEP 525 makes the scheduler requirement explicit. An asynchronous generator cannot finish operations containing await merely because ordinary synchronous code has called a method on it. The event loop must drive the awaitable operations, including operations involved in finalization.

An asynchronous generator also differs from a coroutine that returns one result. A nonempty return statement is a syntax error in an asynchronous generator. Its result is a sequence of yielded values, and exhaustion has its own protocol signal. These distinctions help explain why an asynchronous function with yield produces a different kind of object from an asynchronous function without it.

I prefer the generator form for producers whose state follows naturally from a loop. A manually implemented iterator can still be appropriate, but its fields and methods should express a requirement that the function form cannot express as clearly. The new syntax removes state bookkeeping when the generator's suspended frame already represents that state.

Early termination makes cleanup part of the design

A producer can hold a resource while it is suspended at yield. PEP 525 illustrates the problem with a database transaction that contains an asynchronous iteration over a cursor. If the consumer breaks out of its loop, the producer may still need to execute the cleanup associated with that transaction. That cleanup may contain an asynchronous operation.

The generator provides aclose() for this case. It returns an awaitable that injects GeneratorExit into the suspended generator and drives its termination. Calling the method without awaiting its result does not perform the complete asynchronous closing operation. This follows the same scheduler requirement as ordinary iteration.

The proposal permits await during a finally block but rejects a yield there during finalization with RuntimeError. Cleanup can wait for its work to finish. It cannot respond to closing by delivering another ordinary item to a consumer that has already requested termination. This makes the finalization path distinct from the path that produces more values.

The interpreter and event loop cooperate through sys.set_asyncgen_hooks(). The hooks include a first-iteration callback and a finalizer callback. A generator caches the applicable finalizer when iteration starts, which allows finalization to refer to the event loop responsible for its execution. The hooks are thread-specific, so their configuration belongs to the thread running that loop.

For asyncio, the design includes loop.shutdown_asyncgens(). The event loop tracks asynchronous generators and uses that coroutine to close those still open during shutdown. The PEP's shutdown example drives this operation before closing the loop. Closing a scheduler while generators still require it for cleanup would remove the mechanism needed to finish their work.

This is the part of the release that interests me more than reducing the number of iterator methods. Python has made a resource lifetime visible inside an ordinary function body, but asynchronous resource lifetimes still depend on the execution environment. That dependency remains part of the program's design even when the producer itself becomes shorter.

Annotations keep a separate contract

The release also includes the provisional variable annotation syntax specified by PEP 526. It permits annotations on module and class variables as well as local variables. An annotation can appear without an initial value, which avoids introducing a placeholder merely to describe the intended type.

The interpreter does not enforce an annotated type on assignment. Module and class annotations can evaluate annotation expressions and store information for simple names, while local variable annotations are not evaluated or retained on every call. This distinction matters for code that inspects annotations at runtime and for code whose annotations reference other names.

An annotation without an assignment also leaves the variable without a value. Inside a function it establishes a local name, so reading that name before assigning it raises UnboundLocalError. The syntax describes a variable's intended use without supplying an initialization policy. A program must still choose when it has enough information to construct the value.

Together, these changes give Python 3.6 several ways to express operations with less manual machinery. Their guarantees remain specific to their protocols. Formatting calls an object's formatting behavior, asynchronous iteration requires awaitable progress, and annotations provide metadata for tools. Code using the new syntax still depends on those individual rules.