UPDATE: use Balthazar’s answer for Celery 3.1+ @task(bind=True) def do_job(self, path): cache.set(self.request.id, operation_results) Feel free to upvote his answer. Old answer: Since Celery 2.2.0, information related to the currently executed task is saved to task.request (it’s called «the context»). So you should get task id from this context (not from keyword arguments, which are deprecated): … Read more
Let’s distinguish between workers and worker processes. You spawn a celery worker, this then spawns a number of processes (depending on things like –concurrency and –autoscale, the default is to spawn as many processes as cores on the machine). There is no point in running more than one worker on a particular machine unless you … Read more
Your goals are: Distribute your work to many machines (distributed computing/distributed parallel processing) Distribute the work on a given machine across all CPUs (multiprocessing/threading) Celery can do both of these for you fairly easily. The first thing to understand is that each celery worker is configured by default to run as many tasks as there … Read more
Have you tried just running the task from the Django shell? You can use the .apply method of a task to ensure that it is run eagerly and locally. Assuming the task is called my_task in Django app myapp in a tasks submodule: $ python manage.py shell >>> from myapp.tasks import my_task >>> eager_result = … Read more
Prefetching can improve the performance. Workers don’t need to wait for the next message from a broker to process. Communicating with a broker once and processing a lot of messages gives a performance gain. Getting a message from a broker (even from a local one) is expensive compared to the local memory access. Workers are … Read more
It depends what you want your tasks to be doing, if you need to distribute them, and how you want to manage them. A crontab is capable of executing a script every N intervals. It runs, and then returns. Essentially you get a single execution each interval. You could just direct a crontab to execute … Read more
You are right, you don’t need Celery at all. When you are designing a distributed system there are a lot of options and there is no right way to do things that fits all situations. Many people find that it is more flexible to have pools of message consumers waiting for a message to appear … Read more
revoke cancels the task execution. If a task is revoked, the workers ignore the task and do not execute it. If you don’t use persistent revokes your task can be executed after worker’s restart. https://docs.celeryq.dev/en/stable/userguide/workers.html#worker-persistent-revokes revoke has an terminate option which is False by default. If you need to kill the executing task you need … Read more
Here is what I have found while trying to answer this exact same question. It’s probably not comprehensive, and may even be inaccurate on some points. In short, RQ is designed to be simpler all around. Celery is designed to be more robust. They are both excellent. Documentation. RQ’s documentation is comprehensive without being complex, … Read more
Return the task_id (which is given from .delay()) and ask the celery instance afterwards about the state: x = method.delay(1,2) print x.task_id When asking, get a new AsyncResult using this task_id: from celery.result import AsyncResult res = AsyncResult(“your-task-id”) res.ready()