log.retention.hours is a property of a broker which is used as a default value when a topic is created. When you change configurations of currently running topic using kafka-topics.sh, you should specify a topic-level property. A topic-level property for log retention time is retention.ms. From Topic-level configuration in Kafka 0.8.1 documentation: Property: retention.ms Default: 7 … Read more
To increase the number of replicas for a given topic you have to: 1. Specify the extra replicas in a custom reassignment json file For example, you could create increase-replication-factor.json and put this content in it: {“version”:1, “partitions”:[ {“topic”:”signals”,”partition”:0,”replicas”:[0,1,2]}, {“topic”:”signals”,”partition”:1,”replicas”:[0,1,2]}, {“topic”:”signals”,”partition”:2,”replicas”:[0,1,2]} ]} 2. Use the file with the –execute option of the kafka-reassign-partitions tool [or … Read more
Use kafka-consumer-groups.sh For example bin/kafka-consumer-groups.sh –list –bootstrap-server localhost:9092 bin/kafka-consumer-groups.sh –describe –group mygroup –bootstrap-server localhost:9092
You can use console consumer to view messages produced on some topic: bin/kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic test –from-beginning
Kafka and ActiveMQ may have some overlaps but they were originally designed for different purposes. So comparing them is just like comparing an Apple and an Orange. Kafka Kafka is a distributed streaming platform with very good horizontal scaling capability. It allows applications to process and re-process streamed data on disk. Due to it’s high … Read more
It depends on Group ID. Suppose you have a topic with 12 partitions. If you have 2 Kafka consumers with the same Group Id, they will both read 6 partitions, meaning they will read different set of partitions = different set of messages. If you have 4 Kafka cosnumers with the same Group Id, each … Read more
This command will give you the list of the active brokers between brackets: ./bin/zookeeper-shell.sh localhost:2181 ls /brokers/ids
Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don’t already have one. If you do not want to install and have a separate zookeeper server, you can use the convenience script packaged with kafka to get a quick-and-dirty single-node ZooKeeper instance. Starting the single-node Zookeeper instance: bin/zookeeper-server-start.sh config/zookeeper.properties Starting … Read more
As I mentioned here Purge Kafka Queue: Tested in Kafka 0.8.2, for the quick-start example: First, Add one line to server.properties file under config folder: delete.topic.enable=true then, you can run this command: bin/kafka-topics.sh –zookeeper localhost:2181 –delete –topic test
Before KIP-62, there is only session.timeout.ms (ie, Kafka 0.10.0 and earlier). max.poll.interval.ms is introduced via KIP-62 (part of Kafka 0.10.1). KIP-62, decouples heartbeats from calls to poll() via a background heartbeat thread, allowing for a longer processing time (ie, time between two consecutive poll()) than heartbeat interval. Assume processing a message takes 1 minute. If … Read more