with are non-trivial syntactic sugar that encapsulate several steps of calling related methods. This makes it impossible to manually add
awaits between these steps – but properly usable
with need that. At the same time, this means it is vital to have
async support for them.
Why we can’t
await nice things
Python’s statements and expressions are backed by so-called protocols: When an object is used in some specific statement/expression, Python calls corresponding “special methods” on the object to allow customization. For example,
x in [1, 2, 3] delegates to
list.__contains__ to define what
in actually means.
Most protocols are straightforward: There is one special method called for each statement/expression. If the only
async feature we have is the primitive
await, then we can still make all these “one special method” statements/expression “
async” by sprinkling
await at the right place.
In contrast, the
with statements both correspond to multiple steps:
for uses the iterator protocol to repeatedly fetch the
__next__ item of an iterator, and
with uses the context manager protocol to both enter and exit a context.
The important part is that both have more than one step that might need to be asynchronous. While we could manually sprinkle an
await at one of these steps, we cannot hit all of them.
The easier case to look at is
with: we can address at the
We could naively define a syncronous context manager with asynchronous special methods. For entering this actually works by adding an
with AsyncEnterContext() as acm: context = await acm print("I entered an async context and all I got was this lousy", context)
However, it already breaks down if we use a single
withstatement for multiple contexts: We would first enter all contexts at once, then await all of them at once.
with AsyncEnterContext() as acm1, AsyncEnterContext() as acm2: context1, context2 = await acm1, await acm2 # wrong! acm1 must be entered completely before loading acm2 print("I entered many async contexts and all I got was a rules lawyer telling me I did it wrong!")
Worse, there is just no single point where we could
While it’s true that
with are syntactic sugar, they are non-trivial syntactic sugar: They make multiple actions nicer. As a result, one cannot naively
await individual actions of them. Only a blanket
async with and
async for can cover every step.
Why we want to
async nice things
with are abstractions: They fully encapsulate the idea of iteration/contextualisation.
Picking one of the two again, Python’s
for is the abstraction of internal iteration – for contrast, a
while is the abstraction of external iteration. In short, that means the entire point of
for is that the programmer does not have to know how iteration actually works.
- Compare how one would iterate a
some_list = list(range(20)) index = 0 # lists are indexed from 0 while index < len(some_list): # lists are indexed up to len-1 print(some_list[index]) # lists are directly index'able index += 1 # lists are evenly spaced for item in some_list: # lists are iterable print(item)
whileiteration relies on knowledge about how lists work concretely: It pulls implementation details out of the iterable and puts them into the loop. In contrast, internal
foriteration only relies on knowing that lists are iterable. It would work with any implementation of lists, and in fact any implementation of iterables.
Bottom line is the entire point of
for – and
with – is not to bother with implementation details. That includes having to know which steps we need to sprinkle with async. Only a blanket
async with and
async for can cover every step without us knowing which.
Why we need to
async nice things
A valid question is why
async variants, but others do not. There is a subtle point about
with that is not obvious in daily usage: both represent concurrency – and concurrency is the domain of
Without going too much into detail, a handwavy explanation is the equivalence of handling routines (
()), iterables (
for) and context managers (
with). As has been established in the answer cited in the question, coroutines are actually a kind of generators. Obviously, generators are also iterables and in fact we can express any iterable via a generator. The less obvious piece is that context managers are also equivalent to generators – most importantly,
contextlib.contextmanager can translate generators to context managers.
To consistently handle all kinds of concurrency, we need
async variants for routines (
await), iterables (
async for) and context managers (
async with). Only a blanket
async with and
async for can cover every step consistently.