Sorry to answer my own question, but I’ve found at least a workaround so in case anyone else has a similar issue I want to post it here. I’ll accept any better answers out there.
I believe the root of the issue is http://bugs.python.org/issue9400 . This tells me two things:
- I’m not crazy, what I’m trying to do really is supposed to work
- At least in python2, it is very difficult if not impossible to pickle ‘exceptions’ back to the parent process. Simple ones work, but many others don’t.
In my case, my worker function was launching a subprocess that was segfaulting. This returned CalledProcessError exception, which is not pickleable. For some reason, this makes the pool object in the parent go out to lunch and not return from the call to join().
In my particular case, I don’t care what the exception was. At most I want to log it and keep going. To do this, I simply wrap my top worker function in a try/except clause. If the worker throws any exception, it is caught before trying to return to the parent process, logged, and then the worker process exits normally since it’s no longer trying to send the exception through. See below:
def process_file_wrapped(filenamen, foo, bar, baz=biz):
try:
process_file(filename, foo, bar, baz=biz)
except:
print('%s: %s' % (filename, traceback.format_exc()))
Then, I have my initial map function call process_file_wrapped() instead of the original one. Now my code works as intended.