Why do we need `async for` and `async with`?

by

TLDR: for and with are non-trivial syntactic sugar that encapsulate several steps of calling related methods. This makes it impossible to manually add awaits between these steps – but properly usable async for/with need that. At the same time, this means it is vital to have async support for them.

Why we can’t await nice things

Python’s statements and expressions are backed by so-called protocols: When an object is used in some specific statement/expression, Python calls corresponding “special methods” on the object to allow customization. For example, x in [1, 2, 3] delegates to list.__contains__ to define what in actually means.
Most protocols are straightforward: There is one special method called for each statement/expression. If the only async feature we have is the primitive await, then we can still make all these “one special method” statements/expression “async” by sprinkling await at the right place.

In contrast, the for and with statements both correspond to multiple steps: for uses the iterator protocol to repeatedly fetch the __next__ item of an iterator, and with uses the context manager protocol to both enter and exit a context.
The important part is that both have more than one step that might need to be asynchronous. While we could manually sprinkle an await at one of these steps, we cannot hit all of them.

  • The easier case to look at is with: we can address at the __enter__ and __exit__ method separately.

    We could naively define a syncronous context manager with asynchronous special methods. For entering this actually works by adding an await strategically:

    with AsyncEnterContext() as acm:
    context = await acm
    print("I entered an async context and all I got was this lousy", context)

    However, it already breaks down if we use a single with statement for multiple contexts: We would first enter all contexts at once, then await all of them at once.

    with AsyncEnterContext() as acm1, AsyncEnterContext() as acm2:
    context1, context2 = await acm1, await acm2  # wrong! acm1 must be entered completely before loading acm2
    print("I entered many async contexts and all I got was a rules lawyer telling me I did it wrong!")

    Worse, there is just no single point where we could await exiting properly.

While it’s true that for and with are syntactic sugar, they are non-trivial syntactic sugar: They make multiple actions nicer. As a result, one cannot naively await individual actions of them. Only a blanket async with and async for can cover every step.

Why we want to async nice things

Both for and with are abstractions: They fully encapsulate the idea of iteration/contextualisation.

Picking one of the two again, Python’s for is the abstraction of internal iteration – for contrast, a while is the abstraction of external iteration. In short, that means the entire point of for is that the programmer does not have to know how iteration actually works.

  • Compare how one would iterate a list using for or while: 
    some_list = list(range(20))
index = 0                      # lists are indexed from 0
while index < len(some_list):  # lists are indexed up to len-1
    print(some_list[index])    # lists are directly index'able
    index += 1                 # lists are evenly spaced

for item in some_list:         # lists are iterable
    print(item)

    The external while iteration relies on knowledge about how lists work concretely: It pulls implementation details out of the iterable and puts them into the loop. In contrast, internal for iteration only relies on knowing that lists are iterable. It would work with any implementation of lists, and in fact any implementation of iterables.

Bottom line is the entire point of for – and with – is not to bother with implementation details. That includes having to know which steps we need to sprinkle with async. Only a blanket async with and async for can cover every step without us knowing which.

Why we need to async nice things

A valid question is why for and with get async variants, but others do not. There is a subtle point about for and with that is not obvious in daily usage: both represent concurrency – and concurrency is the domain of async.

Without going too much into detail, a handwavy explanation is the equivalence of handling routines (()), iterables (for) and context managers (with). As has been established in the answer cited in the question, coroutines are actually a kind of generators. Obviously, generators are also iterables and in fact we can express any iterable via a generator. The less obvious piece is that context managers are also equivalent to generators – most importantly, contextlib.contextmanager can translate generators to context managers.

To consistently handle all kinds of concurrency, we need async variants for routines (await), iterables (async for) and context managers (async with). Only a blanket async with and async for can cover every step consistently.

Leave a Comment