The Flask Celery Based Background Tasks page (http://flask.pocoo.org/docs/patterns/celery/) suggests this to start celery: celery -A your_application worker The your_application string has to point to your application’s package or module that creates the celery object. Assuming the code resides in application.py, explicitly pointing to the celery object (not just the module name) avoided the error: celery … Read more
It works using AsyncResult. (see this answer) So first create the task: from cel.tasks import add res = add.delay(3,4) print(res.status) # ‘SUCCESS’ print(res.id) # ‘432890aa-4f02-437d-aaca-1999b70efe8d’ Then start another python shell: from celery.result import AsyncResult from cel.tasks import app res = AsyncResult(‘432890aa-4f02-437d-aaca-1999b70efe8d’,app=app) print(res.state) # ‘SUCCESS’ print(res.get()) # 7
Unacknowledged messages are those which have been delivered across the network to a consumer but have not yet been ack’ed or rejected — but that consumer hasn’t yet closed the channel or connection over which it originally received them. Therefore the broker can’t figure out if the consumer is just taking a long time to … Read more
I like tigeronk2’s idea of one connection per worker. As he says, Celery maintains its own pool of workers so there really isn’t a need for a separate database connection pool. The Celery Signal docs explain how to do custom initialization when a worker is created so I added the following code to my tasks.py … Read more
You can run flower with –auth flag, which will authenticate using a particular google email: celery flower –auth=your.email@gmail.com Edit 1: New version of Flower requires couple more flags and a registered OAuth2 Client with Google Developer Console: celery flower \ –auth=your.email@gmail.com \ –oauth2_key=”client_id” \ –oauth2_secret=”client_secret” \ –oauth2_redirect_uri=”http://example.com:5555/login” oauth2_redirect_uri has to be the actual flower login … Read more
Adding the following lines to cloud/celery.py: import celery print celery.__file__ gave me the file itself and not the celery module from the library. After renaming celery.py to celeryapp.py and adjusting the imports all errors were gone. Note: That leads to a change in starting the worker: celery worker –app=cloud.celeryapp:app For those running celery==3.1.2 and getting … Read more
Finally I found the solution. These posts helped to figure this out. RabbitMQ on EC2 Consuming Tons of CPU and https://serverfault.com/questions/337982/how-do-i-restart-rabbitmq-after-switching-machines What happened was rabbitmq was holding on to all the results that were never freed to the point it became overloaded. I cleared all the stale data in /var/lib/rabbitmq/mnesia/rabbit/, restarted rabbit and it works … Read more
To answer your opening questions: As of version 2.0, Celery provides an easy way to start tasks from other tasks. What you are calling “secondary tasks” are what it calls “subtasks”. See the documentation for Sets of tasks, Subtasks and Callbacks, which @Paperino was kind enough to link to. For version 3.0, Celery changed to … Read more
Update Jan 2022: This answer is outdated. As suggested in comments, please refer to this link The problem is that you are trying to connect to a local instance of RabbitMQ. Look at this line in your settings.py BROKER_URL = ‘amqp://guest:guest@localhost:5672/’ If you are working currently on development, you could avoid setting up Rabbit and … Read more