Even though Kafka provides convenience methods to persist the committed offsets for a given consumer group, you’re not required to use that behavior and can write your own if you feel the need. Even so, the use of Kafka the way you’ve described it is a bit awkward for the use case as each client … Read more
There are 3 possibilities: Increase request.timeout.ms – this is the time that Kafka will wait for whole batch to be ready in buffer. So in your case if there are less than 100 000 messages in buffer, timeout will occur. More info here: https://stackoverflow.com/a/34794261/2707179 Decrease batch-size – related to previous point, it will send batches … Read more
“metadata.broker.list” is deprecated and “bootstrap.servers” is used instead. Removing “metadata.broker.list” configuration from your code should resolve the problem. Please find more details at difference between bootstrap.servers and metadata.broker.list
You can use the excellent kafkacat tool. Sample command: kafkacat -b kafka-broker:9092 -t my_topic_name -C \ -f ‘\nKey (%K bytes): %k Value (%S bytes): %s Timestamp: %T Partition: %p Offset: %o Headers: %h\n’ Sample output: Key (-1 bytes): Value (13 bytes): {foo:”bar 5″} Timestamp: 1548350164096 Partition: 0 Offset: 34 Headers: __connect.errors.topic=test_topic_json,__connect.errors.partition=0,__connect.errors.offset=94,__connect.errors.connector.name=file_sink_03,__connect.errors.task.id=0,__connect.errors.stage=VALU E_CONVERTER,__connect.errors.class.name=org.apache.kafka.connect.json.JsonConverter,__connect.errors.exception.class.name=org.apache.kafka.connect.errors.DataException,__connect.errors.exception.message=Co nverting byte[] to … Read more
Looking at the documentation you linked to it looks like the main difference between the Future and the Callback lies in who initiates the “request is finished, what now?” question. Let’s say we have a customer C and a baker B. And C is asking B to make him a nice cookie. Now there are … Read more
I was having this same problem today. I’m a newbie at Kafka and was simply trying to get a sample Java producer and consumer running. I was able to get the consumer working, but kept getting the same “topic not present in metadata” error as you, with the producer. Finally, out of desperation, I added … Read more
Heartbeats are the basic mechanism to check if all consumers are still up and running. If you get a heartbeat failure because the group is rebalancing, it indicates that your consumer instance took too long to send the next heartbeat and was considered dead and thus a rebalance got triggered. If you want to prevent … Read more
The high watermark indicates the offset of messages that are fully replicated, while the end-of-log offset might be larger if there are newly appended records to the leader partition which are not replicated yet. Consumers can only consume messages up to the high watermark. See this blog post for more details: http://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/
When you are starting your Kafka broker you can define set of properties in conf/server.properties file. This file is just key value property file. One of the properties is auto.create.topics.enable, if it’s set to true (by default) Kafka will create topics automatically when you send messages to non-existing topics. All config options you can find … Read more
I found out the solution after some research and the solution is here. kafka-console-producer command kafka-console-producer.sh –broker-list localhost:9092 –topic topic-name –property “parse.key=true” –property “key.separator=:” After running this command you will enter in producer console and from there you can send key, value messages. For example key1:value1 key2:value2 key3:value3 For more clarity, I am providing sample … Read more