Python based asynchronous workflow modules : What is difference between celery workflow and luigi workflow?
(I’m the author of Luigi) Luigi is not meant for synchronous low-latency framework. It’s meant for large batch processes that run for hours or days. So I think for your use case, Celery might actually be slightly better