Yes, Zookeeper is required for running Kafka. From the Kafka Getting Started documentation: Step 2: Start the server Kafka uses zookeeper so you need to first start a zookeeper server if you don’t already have one. You can use the convenience script packaged with kafka to get a quick-and-dirty single-node zookeeper instance. As to why, … Read more
When structuring your data for Kafka it really depends on how it´s meant to be consumed. In my mind, a topic is a grouping of messages of a similar type that will be consumed by the same type of consumer so in the example above, I would just have a single topic and if you´ll … Read more
It could be related to advertised.host.name setting in your server.properties. What could happen is that your producer is trying to find out who is the leader for a given partition, figures out its advertised.host.name and advertised.port and tries to connect. If these settings are not configured correctly it then may think that the leader is … Read more
It is a bit more complex than you described. The auto.offset.reset config kicks in ONLY if your consumer group does not have a valid offset committed somewhere (2 supported offset storages now are Kafka and Zookeeper), and it also depends on what sort of consumer you use. If you use a high-level java consumer then … Read more
Temporarily update the retention time on the topic to one second: kafka-topics.sh \ –zookeeper <zkhost>:2181 \ –alter \ –topic <topic name> \ –config retention.ms=1000 And in newer Kafka releases, you can also do it with kafka-configs –entity-type topics kafka-configs.sh \ –zookeeper <zkhost>:2181 \ –entity-type topics \ –alter \ –entity-name <topic name> \ –add-config retention.ms=1000 then … Read more
I am one of the original authors of Kafka. Kafka will work very well as a log for event sourcing. It is fault-tolerant, scales to enormous data sizes, and has a built in partitioning model. We use it for several use cases of this form at LinkedIn. For example our open source stream processing system, … Read more
This post already has answers, but I am adding my view with a few pictures from Kafka Definitive Guide Before answering the questions, let’s look at an overview of producer components: 1. When a producer is producing a message – It will specify the topic it wants to send the message to, is that right? … Read more
RabbitMQ is a solid, general-purpose message broker that supports several protocols such as AMQP, MQTT, STOMP, etc. It can handle high throughput. A common use case for RabbitMQ is to handle background jobs or long-running task, such as file scanning, image scaling or PDF conversion. RabbitMQ is also used between microservices, where it serves as … Read more