Note: This answer is outdated as of Celery 3.0, where you now use get_task_logger() to get your per-task logger set up. Please see the Logging section of the What’s new in Celery 3.0 document for details. Celery has dedicated support for logging, per task. See the Task documentation on the subject: You can use the … Read more
I’ve used both recently (2017-2018), and they are both super stable with Celery 4. So your choice can be based on the details of your hosting setup. If you must use Celery version 2 or version 3, go with RabbitMQ. Otherwise… If you are using Redis for any other reason, go with Redis If you … Read more
Just a small addition to other answers. As already stated, bound tasks have access to the task instance. One use case when this is needed are retries: @celery.task(bind=True, max_retries=5) def retrying(self): try: return 1/0 except Exception: self.retry(countdown=5) Another use case is when you want to define custom states for your tasks and be able to … Read more
I would say that Gearman is better for queuing “jobs” and RabbitMQ is better for queuing “data”. Of course, they are both really the same thing, but the way it works out for me is that if you are trying to “fan out” work to be done, and the workers can work independently, Gearman is … Read more
All string values are limited to 512 MiB. This is the size limit you probably care most about. EDIT: Because keys in Redis are strings, the maximum key size is 512 MiB. The maximum number of keys is 2^32 – 1 = 4,294,967,295. Values, on the other hand, can vary in size depending on their … Read more
These technologies aren’t as similar as they initially seem. They address different portions of the application stack and are actually complementary. Gunicorn is for scaling web request concurrency, while celery should be thought of as a worker queue. We’ll get to kubernetes soon. Gunicorn Web request concurrency is primarily limited by network I/O or “I/O … Read more
When your logger initialized in the beginning of “another module” it links to another logger. Which handle your messages. It can be root logger, or usually I see in Django projects – logger with name ”. Best way here, is overriding your logging config: LOGGING = { ‘version’: 1, ‘disable_existing_loggers’: True, ‘formatters’: { ‘simple’: { … Read more
Just keep reading tutorial. It will be explained in Keep Results chapter. To start Celery you need to provide just broker parameter, which is required to send messages about tasks. If you want to retrieve information about state and results returned by finished tasks you need to set backend parameter. You can find full list … Read more
I like to use the override_settings decorator on tests which need celery results to complete. from django.test import TestCase from django.test.utils import override_settings from myapp.tasks import mytask class AddTestCase(TestCase): @override_settings(CELERY_EAGER_PROPAGATES_EXCEPTIONS=True, CELERY_ALWAYS_EAGER=True, BROKER_BACKEND=’memory’) def test_mytask(self): result = mytask.delay() self.assertTrue(result.successful()) If you want to apply this to all tests you can use the celery test runner as … Read more
The task.request.retries attribute contains the number of tries so far, so you can use this to implement exponential back-off: from celery.task import task @task(bind=True, max_retries=3) def update_status(self, auth, status): try: Twitter(auth).update_status(status) except Twitter.WhaleFail as exc: raise self.retry(exc=exc, countdown=2 ** self.request.retries) To prevent a Thundering Herd Problem, you may consider adding a random jitter to your … Read more