How to Guarantee Message delivery with Celery?
A lot has changed since the OP! There is now an option for high-availability aka “mirrored” queues. This goes pretty far toward solving the problem you described. See http://www.rabbitmq.com/ha.html.
message-queue
How does consumer rebalancing work in Kafka?
Depends on what you mean by “blocked”. If you mean “are existing connections closed when rebalance is triggered” then the answer is yes. The current Kafka’s rebalancing algorithm is unfortunately imperfect. Here is what is happening during consumer rebalance. Assume we have a topic with 10 partitions (0-9), and one consumer (lets name it consumer1) … Read more
Messaging Confusion: Pub/Sub vs Multicast vs Fan Out
I’m confused by your choice of three terms to compare. Within RabbitMQ, Fanout and Direct are exchange types. Pub-Sub is a generic messaging pattern but not an exchange type. And you didn’t even mention the 3rd and most important Exchange type, namely Topic. In fact, you can implement Fanout behavior on a Topic exchange just … Read more
Publish/Subscribe vs Producer/Consumer?
There is a difference between the publish/subscribe and producer/consumer models. Publish/Subscriber: Subscribers subscribe to the publisher. Each message the Publisher publishes is sent to all the subscribers. That is, all subscribers receive the same message. (Think of a newspaper or magazine subscription. All subscribers receive the same magazine or newspaper) Producer/Consumer: Each message the producer … Read more
In which domains are message oriented middleware like AMQP useful?
This is a great question. The main uses of messaging are: scaling, offloading work, integration, monitoring, event handling, routing, networking, push, mobility, buffering, queueing, task sharing, alerts, management, logging, batch, data delivery, pubsub, multicast, audit, scheduling, … and more. Basically: anything where you need data but don’t want to make a database request. (Caching is … Read more
difference between pub-sub and push-pull pattern in zeroMq
The difference is that a PUB socket sends the same message to all subscribers, whereas PUSH does a round-robin amongst all its connected PULL sockets. In your example, if you send just a single message from the root, then all the subscribers will receive it (barring slow subscribers, etc.) but only 1 worker. The pub/sub … Read more
Rabbitmq or Gearman – choosing a jobs queue [closed]
I would say that Gearman is better for queuing “jobs” and RabbitMQ is better for queuing “data”. Of course, they are both really the same thing, but the way it works out for me is that if you are trying to “fan out” work to be done, and the workers can work independently, Gearman is … Read more
Performance comparison between ZeroMQ, RabbitMQ and Apache Qpid
RabbitMQ is probably doing persistence on those messages. I think you need to set the message priority or another option in messages to not do persistence. Performance will improve 10x then. You should expect at least 100K messages/second through an AMQP broker. In OpenAMQ we got performance up to 300K messages/second. AMQP was designed for … Read more
Message Queue vs Message Bus — what are the differences?
Message Bus A Message Bus is a messaging infrastructure to allow different systems to communicate through a shared set of interfaces(message bus). Source: EIP Message Queue The basic idea of a message queue is a simple one: Two (or more) processes can exchange information via access to a common system message queue. The sending process … Read more