Python Asynchronous Comprehensions – how do they work?

You are basically asking how an async for loop works over a regular loop. That you can now use such a loop in a list comprehension doesn’t make any difference here; that’s just an optimisation that avoids repeated list.append() calls, exactly like a normal list comprehension does.

An async for loop then, simply awaits each next step of the iteration protocol, where a regular for loop would block.

To illustrate, imagine a normal for loop:

for foo in bar:
    ...

For this loop, Python essentially does this:

bar_iter = iter(bar)
while True:
    try:
        foo = next(bar_iter)
    except StopIteration:
        break
    ...

The next(bar_iter) call is not asynchronous; it blocks.

Now replace for with async for, and what Python does changes to:

bar_iter = aiter(bar)  # aiter doesn't exist, but see below
while True:
    try:
        foo = await anext(bar_iter)  # anext doesn't exist, but see below
    except StopIteration:
        break
    ...

In the above example aiter() and anext() are fictional functions; these are functionally exact equivalents of their iter() and next() brethren but instead of __iter__ and __next__ these use __aiter__ and __anext__. That is to say, asynchronous hooks exist for the same functionality but are distinguished from their non-async variants by the prefix a.

The await keyword there is the crucial difference, so for each iteration an async for loop yields control so other coroutines can run instead.

Again, to re-iterate, all this already was added in Python 3.5 (see PEP 492), all that is new in Python 3.6 is that you can use such a loop in a list comprehension too. And in generator expressions and set and dict comprehensions, for that matter.

Last but not least, the same set of changes also made it possible to use await <expression> in the expression section of a comprehension, so:

[await func(i) for i in someiterable]

is now possible.