You’re creating new processes inside a loop, and then forgetting to close them once you’re done with them. As a result, there comes a point where you have too many open processes. This is a bad idea. You could fix this by using a context manager which automatically calls pool.terminate, or manually call pool.terminate yourself. … Read more
The simplest way would probably be to apply tqdm() around the inputs, rather than the mapping function. For example: inputs = zip(param1, param2, param3) with mp.Pool(8) as pool: results = pool.starmap(my_function, tqdm.tqdm(inputs, total=len(param1))) Note that the bar is updated when my_function is called, rather than when it returns. If that distinction matters, you can consider … Read more
Error 1: AttributeError: Can’t pickle local object ‘SomeClass.some_method..single’ You solved this error yourself by moving the nested target-function single() out to the top-level. Background: Pool needs to pickle (serialize) everything it sends to its worker-processes (IPC). Pickling actually only saves the name of a function and unpickling requires re-importing the function by name. For that … Read more