We had to build our own job queue system to meet requirements similar to yours ( UI thread must always respond within 33ms, jobs can run from 15-15000ms ), because there really was nothing out there that quite met our needs, let alone was performant.
Unfortunately our code is about as proprietary as proprietary gets, but I can give you some of the most salient features:
- We start up one thread per core at the beginning of the program. Each pulls work from a global job queue. Jobs consist of a function object and a glob of associated data (really an elaboration on a func_ptr and void *). Thread 0, the fast client loop, isn’t allowed to work on jobs, but the rest grab as they can.
- The job queue itself ought to be a lockless data structure, such as a lock-free singly linked list (Visual Studio comes with one). Avoid using a mutex; contention for the queue is surprisingly high, and grabbing mutexes is costly.
-
Pack up all the necessary data for the job into the job object itself — avoid having pointer from the job back into the main heap, where you’ll have to deal with contention between jobs and locks and all that other slow, annoying stuff. For example, all the simulation parameters should go into the job’s local data blob. The results structure obviously needs to be something that outlives the job: you can deal with this either by a) hanging onto the job objects even after they’ve finished running (so you can use their contents from the main thread), or b) allocating a results structure specially for each job and stuffing a pointer into the job’s data object. Even though the results themselves won’t live in the job, this effectively gives the job exclusive access to its output memory so you needn’t muss with locks.
-
Actually I’m simplifying a bit above, since we need to choreograph exactly which jobs run on which cores, so each core gets its own job queue, but that’s probably unnecessary for you.