If you increase max.poll.interval.ms that says “it’s ok to spend time processing a large batch of records” and you’ll gain throughput if you can process larger batches more efficiently than smaller ones. To decrease max.poll.records says ”take fewer records so there’s enough time to process them” and would favor latency over throughput. Also consider that … Read more
This exception is occurring because you are queueing records at a much faster rate than they can be sent. When you call the send method, the ProducerRecord will be stored in an internal buffer for sending to the broker. The method returns immediately once the ProducerRecord has been buffered, regardless of whether it has been … Read more
Heartbeats are the basic mechanism to check if all consumers are still up and running. If you get a heartbeat failure because the group is rebalancing, it indicates that your consumer instance took too long to send the next heartbeat and was considered dead and thus a rebalance got triggered. If you want to prevent … Read more
Not as far as I know. The trick I used when I needed to do something similar was: setup a virtual interface for each Kafka broker open a tunnel to each broker so that broker n is bound to virtual interface n configure your /etc/hosts file so that the advertised hostname of broker n is … Read more
In fact, each consumer belongs to a consumer group. When Kafka cluster sends data to a consumer group, all records of a partition will be sent to a single consumer in the group. If there’re more paritions than consumers in a group, some consumers will consume data from more than one partition. If there’re more … Read more
which broker should my producer use? Do I need to manually switch the broker to balance the load? Kafka runs on cluster, meaning set of nodes, so while producing anything you need to tell him the LIST of brokers that you’ve configured for your application, below is a small note taken from their documentation. “metadata.broker.list” … Read more
Instead of changing server.properties include the address 0.0.0.0 in the code itself. Instead of /usr/bin/kafka-console-producer –broker-list Hostname:9092 –topic MyFirstTopic1 use /usr/bin/kafka-console-producer –broker-list 0.0.0.0:9092 –topic MyFirstTopic1
Kafka and zookeeper are different things. If you are running both on the same machine, you need to open both ports, of corse. kafka default ports: 9092, can be changed on server.properties; zookeeper default ports: 2181 for client connections; 2888 for follower(other zookeeper nodes) connections; 3888 for inter nodes connections; That’s it. Kafka, also has … Read more
Since Kafka 2.5.0 (see https://issues.apache.org/jira/browse/KAFKA-9040), you can now use the –all option to see all (default and overridden) topic configuration: % kafka-configs –bootstrap-server <KAFKA_SERVERS> –entity-type topics –entity-name <TOPIC_NAME> –describe –all All configs for topic <TOPIC_NAME> are: compression.type=producer sensitive=false synonyms={} message.format.version=1.0-IV0 sensitive=false synonyms={} file.delete.delay.ms=60000 sensitive=false synonyms={} leader.replication.throttled.replicas= sensitive=false synonyms={} max.message.bytes=1000012 sensitive=false synonyms={} min.compaction.lag.ms=0 sensitive=false synonyms={} message.timestamp.type=CreateTime … Read more
Update Since Apache Kafka 2.1, offsets won’t be deleted as long as the consumer group is active, independent if the consumers commit offsets or not, ie, the offset.retention.minutes clocks only starts to tick when the group becomes empty (in older released, the clock started to tick directly when the commit happened). Cf. https://cwiki.apache.org/confluence/display/KAFKA/KIP-211%3A+Revise+Expiration+Semantics+of+Consumer+Group+Offsets Original Answer … Read more