How to set up celery workers on separate machines?

My understanding is that your app will push requests into a queueing system (e.g. rabbitMQ) and then you can start any number of workers on different machines (with access to the same code as the app which submitted the task). They will pick out tasks from the message queue and then get to work on them. Once they’re done, they will update the tombstone database.

The upshot of this is that you don’t have to do anything special to start multiple workers. Just start them on separate identical (same source tree) machines.

The server which has the message queue need not be the same as the one with the workers and needn’t be the same as the machines which submit jobs. You just need to put the location of the message queue in your celeryconfig.py and all the workers on all the machines can pick up jobs from the queue to perform tasks.

Leave a Comment

Hata!: SQLSTATE[HY000] [1045] Access denied for user 'divattrend_liink'@'localhost' (using password: YES)