Try Dequeue in ConcurrentQueue
Isn’t this what BlockingCollection is designed for? As I understand it, you can wrap your ConcurrentQueue with one of these, and then call Take.
producer-consumer
C# producer/consumer
The code is older than that – I wrote it some time before .NET 2.0 came out. The concept of a producer/consumer queue is way older than that though 🙂 Yes, that code is safe as far as I’m aware – but it has some deficiencies: It’s non-generic. A modern version would certainly be generic. … Read more
How does consumer rebalancing work in Kafka?
Depends on what you mean by “blocked”. If you mean “are existing connections closed when rebalance is triggered” then the answer is yes. The current Kafka’s rebalancing algorithm is unfortunately imperfect. Here is what is happening during consumer rebalance. Assume we have a topic with 10 partitions (0-9), and one consumer (lets name it consumer1) … Read more
Job queue as SQL table with multiple consumers (PostgreSQL)
I use postgres for a FIFO queue as well. I originally used ACCESS EXCLUSIVE, which yields correct results in high concurrency, but has the unfortunate effect of being mutually exclusive with pg_dump, which acquires a ACCESS SHARE lock during its execution. This causes my next() function to lock for a very long time (the duration … Read more
How to iterate Queue.Queue items in Python?
You can loop over a copy of the underlying data store: for elem in list(q.queue) Eventhough this bypasses the locks for Queue objects, the list copy is an atomic operation and it should work out fine. If you want to keep the locks, why not pull all the tasks out of the queue, make your … Read more
Producer/Consumer threads using a Queue
Java 5+ has all the tools you need for this kind of thing. You will want to: Put all your Producers in one ExecutorService; Put all your Consumers in another ExecutorService; If necessary, communicate between the two using a BlockingQueue. I say “if necessary” for (3) because from my experience it’s an unnecessary step. All … Read more
Blocking queue and multi-threaded consumer, how to know when to stop
You should continue to take() from the queue. You can use a poison pill to tell the worker to stop. For example: private final Object POISON_PILL = new Object(); @Override public void run() { //worker loop keeps taking en element from the queue as long as the producer is still running or as //long as … Read more
Is Zookeeper a must for Kafka? [closed]
Yes, Zookeeper is required for running Kafka. From the Kafka Getting Started documentation: Step 2: Start the server Kafka uses zookeeper so you need to first start a zookeeper server if you don’t already have one. You can use the convenience script packaged with kafka to get a quick-and-dirty single-node zookeeper instance. As to why, … Read more