My understanding is that your app will push requests into a queueing system (e.g. rabbitMQ) and then you can start any number of workers on different machines (with access to the same code as the app which submitted the task). They will pick out tasks from the message queue and then get to work on … Read more
This question was answered on google groups. I AM NOT THE AUTHOR, all credit goes to Jean Mark Here’s a proper solution for this. Confirmed working, In my scenario, I sub-classed Periodic Task and created a model out of it since I can add other fields to the model as I need and also so … Read more
You are coming across this issue, which affects Python 2.7.11 (Kombu is required by Celery). The issue is fixed in Kombu 3.0.30.
Based on MattH’s answer, you could use a decorator like this: from django.core.cache import cache import functools def single_instance_task(timeout): def task_exc(func): @functools.wraps(func) def wrapper(*args, **kwargs): lock_id = “celery-single-instance-” + func.__name__ acquire_lock = lambda: cache.add(lock_id, “true”, timeout) release_lock = lambda: cache.delete(lock_id) if acquire_lock(): try: func(*args, **kwargs) finally: release_lock() return wrapper return task_exc then, use it like … Read more
We sorted by just updating to celery==4.1.1 it seems the latest release for the 4.1.X sorted out the module name change on kombu
EDIT: 01/12/2021 previous answer (find it at the bottom) didn’t age well therefore I added a combination of possible solutions that may satisfy those who still look on how to co-use asyncio and Celery Lets quickly break up the use cases first (more in-depth analysis here: asyncio and coroutines vs task queues): If the task … Read more
You can set task time limits (hard and/or soft) either while defining a task or while calling. from celery.exceptions import SoftTimeLimitExceeded @celery.task(time_limit=20) def mytask(): try: return do_work() except SoftTimeLimitExceeded: cleanup_in_a_hurry() or mytask.apply_async(args=[], kwargs={}, time_limit=30, soft_time_limit=10)
Here’s the code I’ve been using. celery.task.control.Inspect.stats() returns a dict containing lots of details about the currently available workers, None if there are no workers running, or raises an IOError if it can’t connect to the message broker. I’m using RabbitMQ – it’s possible that other messaging systems might behave slightly differently. This worked in … Read more
Don’t use #2 unless you are using celery v3. If you are using celery v4, use #1. Use #3 in instances where you are writing a reusable library or django app. For example, if you are writing an open source set of tasks that allow you to manage aws ec2 instances using celery, you would … Read more