If you are running Celery with the CELERY_ALWAYS_EAGER set to True, then make sure you include this line in your settings too: CELERY_EAGER_PROPAGATES_EXCEPTIONS = True http://docs.celeryproject.org/en/latest/configuration.html#celery-eager-propagates-exceptions
Assuming you have both Python’s celery and django-celery installed, create the following tasks.py file under your app: utils/tasks.py from celery import task # other imports @task() def create_user(data): user = User.objects.create_user( username=data[‘username’], email=None, password=data[‘password’] ) user.save() profile = UserProfile() profile.user = user profile.token = generate_token() profile.save() return None Delete your utils/utilities.py file in your example … Read more
The celery inspect ping command comes in handy, as it does a whole trip: it sends a “ping” task on the broker, workers respond and celery fetches the responses. Assuming your app is named tasks.add, you may ping all your workers: /app $ celery inspect ping -A tasks.add -> celery@aa7c21dd0e96: OK pong -> celery@57615db15d80: OK … Read more
The twisted reactor cannot be restarted. A work around for this is to let the celery task fork a new child process for each crawl you want to execute as proposed in the following post: Running Scrapy spiders in a Celery task This gets around the “reactor cannot be restart-able” issue by utilizing the multiprocessing … Read more
(I’m the author of Luigi) Luigi is not meant for synchronous low-latency framework. It’s meant for large batch processes that run for hours or days. So I think for your use case, Celery might actually be slightly better
Channels in Django are meant for asynchronous handling of requests. The standard model Django uses is Request-Response but that has significant limitations. We cannot do anything outside the restrictions of that model. Channels came about to allow Web Socket support and building complex applications around Web Sockets, so that we can send multiple messages, manage … Read more
Turns out it cant be done at queue level for multiple workers. IT can be done at queue level for 1 worker. Or at queue level for each worker. So if u say 10 jobs/ minute on 5 workers. Your workers will process upto 50 jobs per minute collectively. So to have only 10 jobs … Read more
Okay here is how I got Scrapy working with my Django project that uses Celery for queuing up what to crawl. The actual workaround came primarily from joehillen’s code located here http://snippets.scrapy.org/snippets/13/ First the tasks.py file from celery import task @task() def crawl_domain(domain_pk): from crawl import domain_crawl return domain_crawl(domain_pk) Then the crawl.py file from multiprocessing … Read more
From the Celery FAQ: result = MyTask.AsyncResult(task_id) result.get()
# Retrieve tasks # Reference: http://docs.celeryproject.org/en/latest/reference/celery.events.state.html query = celery.events.state.tasks_by_type(your_task_name) # Kill tasks # Reference: http://docs.celeryproject.org/en/latest/userguide/workers.html#revoking-tasks for uuid, task in query: celery.control.revoke(uuid, terminate=True)