@GP89 mentioned a good solution. Use a queue to send the writing tasks to a dedicated process that has sole write access to the file. All the other workers have read only access. This will eliminate collisions. Here is an example that uses apply_async, but it will work with map too: import multiprocessing as mp … Read more
The best solution for your problem is to utilize a Pool. Using Queues and having a separate “queue feeding” functionality is probably overkill. Here’s a slightly rearranged version of your program, this time with only 2 processes coralled in a Pool. I believe it’s the easiest way to go, with minimal changes to original code: … Read more
Try using multiprocessing.Manager to manage your queue and to also make it accessible to different workers. import multiprocessing def worker(name, que): que.put(“%d is done” % name) if __name__ == ‘__main__’: pool = multiprocessing.Pool(processes=3) m = multiprocessing.Manager() q = m.Queue() workers = pool.apply_async(worker, (33, q))
You can simply use multiprocessing.Pool: from multiprocessing import Pool def process_image(name): sci=fits.open(‘{}.fits’.format(name)) <process> if __name__ == ‘__main__’: pool = Pool() # Create a multiprocessing Pool pool.map(process_image, data_inputs) # process data_inputs iterable with pool
Short Answer Pool’s chunksize-algorithm is a heuristic. It provides a simple solution for all imaginable problem scenarios you are trying to stuff into Pool’s methods. As a consequence, it cannot be optimized for any specific scenario. The algorithm arbitrarily divides the iterable in approximately four times more chunks than the naive approach. More chunks mean … Read more
Do child processes spawned via multiprocessing share objects created earlier in the program? No for Python < 3.8, yes for Python ≥ 3.8. Processes have independent memory space. Solution 1 To make best use of a large structure with lots of workers, do this. Write each worker as a “filter” – reads intermediate results from … Read more
To add to @unutbu’s (not available anymore) and @Henry Gomersall’s answers. You could use shared_arr.get_lock() to synchronize access when needed: shared_arr = mp.Array(ctypes.c_double, N) # … def f(i): # could be anything numpy accepts as an index such another numpy array with shared_arr.get_lock(): # synchronize access arr = np.frombuffer(shared_arr.get_obj()) # no data copying arr[i] = … Read more
After some more googling I found the answer here. It turns out that certain Python modules (numpy, scipy, tables, pandas, skimage…) mess with core affinity on import. As far as I can tell, this problem seems to be specifically caused by them linking against multithreaded OpenBLAS libraries. A workaround is to reset the task affinity … Read more
The multiprocessing.pool.Pool class creates the worker processes in its __init__ method, makes them daemonic and starts them, and it is not possible to re-set their daemon attribute to False before they are started (and afterwards it’s not allowed anymore). But you can create your own sub-class of multiprocesing.pool.Pool (multiprocessing.Pool is just a wrapper function) and … Read more
The problem is that multiprocessing must pickle things to sling them among processes, and bound methods are not picklable. The workaround (whether you consider it “easy” or not;-) is to add the infrastructure to your program to allow such methods to be pickled, registering it with the copy_reg standard library method. For example, Steven Bethard’s … Read more