Suddenly… I found why. According to much more source code of ThreadPoolExecutor: # Workers are created as daemon threads. This is done to allow the interpreter # to exit when there are still idle threads in a ThreadPoolExecutor’s thread # pool (i.e. shutdown() was not called). However, allowing workers to die with # the interpreter … Read more
Overview: There are 2 parts to my answer: Part 1 shows how to gain more speed-up from @niemmi’s ProcessPoolExecutor.map() solution. Part 2 shows when the ProcessPoolExecutor‘s subclasses .submit() and .map() yield non-equivalent compute times. ======================================================================= Part 1: More Speed-up for ProcessPoolExecutor.map() Background: This section builds on @niemmi’s .map() solution, which by itself is excellent. While … Read more
I don’t think this question can be so generically solved; it will depend on each case. From this answer: The more threads you use, the higher concurrency you’ll achieve (up to a point), but the less CPU cycles you’ll get (as there will be context switches). You have to instrument your application under typical workloads … Read more
I want to know how it works internally. as_completed sets up a callback to fire when the future is done, doing so for all the futures it receives. (It uses an internal API equivalent to add_done_callback for this purpose.) When any of the futures completes, as_completed is notified by its callback being run. The callback … Read more
I think I was able to get as far as possible: I changed the _queue_management_worker method in my changed ProcessPoolExecutor module such that the exit code of the failed process is printed: def _queue_management_worker(executor_reference, processes, pending_work_items, work_ids_queue, call_queue, result_queue): “””Manages the communication between this process and the worker processes. … “”” executor = None def … Read more
When using map from concurrent.futures, each element from the iterable is submitted separately to the executor, which creates a Future object for each call. It then returns an iterator which yields the results returned by the futures. Future objects are rather heavyweight, they do a lot of work to allow all the features they provide … Read more
The asyncio documentation covers the differences: class asyncio.Future(*, loop=None) This class is almost compatible with concurrent.futures.Future. Differences: result() and exception() do not take a timeout argument and raise an exception when the future isn’t done yet. Callbacks registered with add_done_callback() are always called via the event loop’s call_soon_threadsafe(). This class is not compatible with the … Read more
It’s kind of painful. Essentially, your worker threads have to be finished before your main thread can exit. You cannot exit unless they do. The typical workaround is to have some global state, that each thread can check to determine if they should do more work or not. Here’s the quote explaining why. In essence, … Read more
You can wrap tqdm around the executor as the following to track the progress: list(tqdm(executor.map(f, iter), total=len(iter)) Here is your example: import time import concurrent.futures from tqdm import tqdm def f(x): time.sleep(0.001) # to visualize the progress return x**2 def run(f, my_iter): with concurrent.futures.ThreadPoolExecutor() as executor: results = list(tqdm(executor.map(f, my_iter), total=len(my_iter))) return results my_iter = … Read more
The problem is that you transform the result of ThreadPoolExecutor.map to a list. If you don’t do this and instead iterate over the resulting generator directly, the results are still yielded in the original order but the loop continues before all results are ready. You can test this with this example: import time import concurrent.futures … Read more