I my MAC box I was facing the same issue of console-consumer not consuming any messages when used the command kafka-console-consumer –bootstrap-server localhost:9095 –from-beginning –topic my-replicated-topic But when I tried with kafka-console-consumer –bootstrap-server localhost:9095 –from-beginning –topic my-replicated-topic –partition 0 It happily lists the messages sent. Is this a bug in Kafka 1.10.11?
The new consumer is also complicated. //assign the topic consumer.assign(); //seek to end of the topic consumer.seekToEnd(); //the position is the latest offset consumer.position();
1. What is the difference? The consumer group coordinator is one of the brokers while the group leader is one of the consumer in a consumer group. The group coordinator is nothing but one of the brokers which receives heartbeats (or polling for messages) from all consumers of a consumer group. Every consumer group has … Read more
Using out of the box console consumer (I am using Kafka 0.9.0.1) you can only print the key and the value of messages using different formats. To print the key, set the property print.key=true. There is another property key.separator that by default is “\t” (a tab) that you can also change to anything you want. … Read more
I found out the solution after some research and the solution is here. kafka-console-producer command kafka-console-producer.sh –broker-list localhost:9092 –topic topic-name –property “parse.key=true” –property “key.separator=:” After running this command you will enter in producer console and from there you can send key, value messages. For example key1:value1 key2:value2 key3:value3 For more clarity, I am providing sample … Read more
You can pipe it in: kafka-console-producer.sh –broker-list localhost:9092 –topic my_topic –new-producer < my_file.txt Found here. From 0.9.0: kafka-console-producer.sh –broker-list localhost:9092 –topic my_topic < my_file.txt
Does it mean if i want to have more than 1 consumer (from the same group) reading from one topic I need to have more than 1 partition? Let’s see the following properties of kafka: each partition is consumed by exactly one consumer in the group one consumer in the group can consume more than … Read more
In Kafka, the responsibility of what has been consumed is the responsibility of the consumer and this is also one of the main reasons why Kafka has such great horizontal scalability. Using the high level consumer API will automatically do this for you by committing consumed offsets in Zookeeper (or a more recent configuration option … Read more
Consumers groups is a Kafka abstraction that enables supporting both point-to-point and publish/subscribe messaging. A consumer can join a consumer group (let us say group_1) by setting its group.id to group_1. Consumer groups is also a way of supporting parallel consumption of the data i.e. different consumers of the same consumer group consume data in … Read more
This works with the 0.9.x consumer. Basically when you create a consumer, you need to assign a consumer group id to this consumer using the property ConsumerConfig.GROUP_ID_CONFIG. Generate the consumer group id randomly every time you start the consumer doing something like this properties.put(ConsumerConfig.GROUP_ID_CONFIG, UUID.randomUUID().toString()); (properties is an instance of java.util.Properties that you will pass … Read more