Short Answer Pool’s chunksize-algorithm is a heuristic. It provides a simple solution for all imaginable problem scenarios you are trying to stuff into Pool’s methods. As a consequence, it cannot be optimized for any specific scenario. The algorithm arbitrarily divides the iterable in approximately four times more chunks than the naive approach. More chunks mean … Read more
No, you don’t, but it’s probably a good idea if you aren’t going to use the pool anymore. Reasons for calling pool.close or pool.join are well said by Tim Peters in this SO post: As to Pool.close(), you should call that when – and only when – you’re never going to submit more work to … Read more
Use shared variable to communicate. For example like this: import multiprocessing def worker(procnum, return_dict): “””worker function””” print(str(procnum) + ” represent!”) return_dict[procnum] = procnum if __name__ == “__main__”: manager = multiprocessing.Manager() return_dict = manager.dict() jobs = [] for i in range(5): p = multiprocessing.Process(target=worker, args=(i, return_dict)) jobs.append(p) p.start() for proc in jobs: proc.join() print(return_dict.values())
There are two key differences between imap/imap_unordered and map/map_async: The way they consume the iterable you pass to them. The way they return the result back to you. map consumes your iterable by converting the iterable to a list (assuming it isn’t a list already), breaking it into chunks, and sending those chunks to the … Read more
Use shared variable to communicate. For example like this: import multiprocessing def worker(procnum, return_dict): “””worker function””” print(str(procnum) + ” represent!”) return_dict[procnum] = procnum if __name__ == “__main__”: manager = multiprocessing.Manager() return_dict = manager.dict() jobs = [] for i in range(5): p = multiprocessing.Process(target=worker, args=(i, return_dict)) jobs.append(p) p.start() for proc in jobs: proc.join() print(return_dict.values())
Here is a list of what can be pickled. In particular, functions are only picklable if they are defined at the top-level of a module. This piece of code: import multiprocessing as mp class Foo(): @staticmethod def work(self): pass if __name__ == ‘__main__’: pool = mp.Pool() foo = Foo() pool.apply_async(foo.work) pool.close() pool.join() yields an error … Read more
is there a variant of pool.map which support multiple arguments? Python 3.3 includes pool.starmap() method: #!/usr/bin/env python3 from functools import partial from itertools import repeat from multiprocessing import Pool, freeze_support def func(a, b): return a + b def main(): a_args = [1,2,3] second_arg = 1 with Pool() as pool: L = pool.starmap(func, [(1, 1), (2, … Read more