pool

How can I monitor/log Tomcat’s thread pool?

by

Direct JMX access Try adding this to catalina.sh/bat: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5005 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false UPDATE: Alex P suggest that the following settings might also be required in some situations: -Dcom.sun.management.jmxremote.host=localhost This enables remote anonymous JMX connections on port 5005. You may also consider JVisualVM which is much more please and allows to browse JMX via plugin. What … Read more

How does the callback function work in multiprocessing map_async?

by

Callback is called once with the result ([[0], [0, 1]]) if you use map_async. >>> from multiprocessing import Pool >>> def myfunc(x): … return [i for i in range(x)] … >>> A = [] >>> def mycallback(x): … print(‘mycallback is called with {}’.format(x)) … A.extend(x) … >>> pool=Pool() >>> r = pool.map_async(myfunc, (1,2), callback=mycallback) >>> … Read more

Can I use a multiprocessing Queue in a function called by Pool.imap?

by

The trick is to pass the Queue as an argument to the initializer. Appears to work with all the Pool dispatch methods. import multiprocessing as mp def f(x): f.q.put(‘Doing: ‘ + str(x)) return x*x def f_init(q): f.q = q def main(): jobs = range(1,6) q = mp.Queue() p = mp.Pool(None, f_init, [q]) results = p.imap(f, … Read more

java.lang.IllegalMonitorStateException: (m=null) Failed to get monitor for

by

See the javadoc for Object.wait. in particular “The current thread must own this object’s monitor.” and “[throws] IllegalMonitorStateException – if the current thread is not the owner of the object’s monitor.” That is, you need to synchronize on the object you are going to call wait on. so your code should be: synchronized (available) { … Read more

How do you pass a Queue reference to a function managed by pool.map_async()?

by

The following code seems to work: import multiprocessing, time def task(args): count = args[0] queue = args[1] for i in xrange(count): queue.put(“%d mississippi” % i) return “Done” def main(): manager = multiprocessing.Manager() q = manager.Queue() pool = multiprocessing.Pool() result = pool.map_async(task, [(x, q) for x in range(10)]) time.sleep(1) while not q.empty(): print q.get() print result.get() … Read more

what is java.io.EOFException, Message: Can not read response from server. Expected to read 4 bytes, read 0 bytes

by

The connection has failed, possibly due to a firewall idle-timeout, etc. If you don’t have your JDBC driver configured to reconnect on failure, then this error will not go away unless you open a new connection. If you are using a database connection pool (you are using one, right?), then you probably want to enable … Read more

Memory usage keep growing with Python’s multiprocessing.pool

by

I had memory issues recently, since I was using multiple times the multiprocessing function, so it keep spawning processes, and leaving them in memory. Here’s the solution I’m using now: def myParallelProcess(ahugearray): from multiprocessing import Pool from contextlib import closing with closing(Pool(15)) as p: res = p.imap_unordered(simple_matching, ahugearray, 100) return res

urllib3 connectionpool – Connection pool is full, discarding connection

by

No data is being lost! The connection is being discarded after the request is completed (because the pool is full, as mentioned). This means that this particular connection is not going to be re-used in the future. Because a urllib3 PoolManager reuses connections, it will limit how many connections are retained per hos to avoid … Read more

multiprocessing.Pool() slower than just using ordinary functions

by

These problems usually boil down to the following: The function you are trying to parallelize doesn’t require enough CPU resources (i.e. CPU time) to rationalize parallelization! Sure, when you parallelize with multiprocessing.Pool(8), you theoretically (but not practically) could get a 8x speed up. However, keep in mind that this isn’t free – you gain this … Read more

Combine Pool.map with shared memory Array in Python multiprocessing

by

Trying again as I just saw the bounty 😉 Basically I think the error message means what it said – multiprocessing shared memory Arrays can’t be passed as arguments (by pickling). It doesn’t make sense to serialise the data – the point is the data is shared memory. So you have to make the shared … Read more