In what situation do we need to use `multiprocessing.Pool.imap_unordered`?

Using pool.imap_unordered instead of pool.imap will not have a large effect on the total running time of your code. It might be a little faster, but not by too much.

What it may do, however, is make the interval between values being available in your iteration more even. That is, if you have operations that can take very different amounts of time (rather than the consistent 0.01 seconds you were using in your example), imap_unordered can smooth things out by yielding faster-calculated values ahead of slower-calculated values. The regular imap will delay yielding the faster ones until after the slower ones ahead of them have been computed (but this does not delay the worker processes moving on to more calculations, just the time for you to see them).

Try making your work function sleep for i*0.1 seconds, shuffling your input list and printing i in your loops. You’ll be able to see the difference between the two imap versions. Here’s my version (the main function and the if __name__ == '__main__' boilerplate was is required to run correctly on Windows):

from multiprocessing import Pool
import time
import random

def work(i):
    time.sleep(0.1*i)
    return i

def main():
    p = Pool(4)
    nums = range(50)
    random.shuffle(nums)

    start = time.time()
    print 'Using imap'
    for i in p.imap(work, nums):
        print i
    print 'Time elapsed: %s' % (time.time() - start)

    start = time.time()
    print 'Using imap_unordered'
    for i in p.imap_unordered(work, nums):
        print i
    print 'Time elapsed: %s' % (time.time() - start)

if __name__ == "__main__":
    main()

The imap version will have long pauses while values like 49 are being handled (taking 4.9 seconds), then it will fly over a bunch of other values (which were calculated by the other processes while we were waiting for 49 to be processed). In contrast, the imap_unordered loop will usually not pause nearly as long at one time. It will have more frequent, but shorter pauses, and its output will tend to be smoother.

Leave a Comment

Hata!: SQLSTATE[HY000] [1045] Access denied for user 'divattrend_liink'@'localhost' (using password: YES)