step 1 : point to a new log directory in server.properties file and save the file log.dirs=C:\Tools\kafka_2.11-2.1.0\kafka-test-logs step 2 : start the kafka server again /bin/windows/kafka-server-start.bat /config/server.properties
When you are starting your Kafka broker you can define set of properties in conf/server.properties file. This file is just key value property file. One of the properties is auto.create.topics.enable, if it’s set to true (by default) Kafka will create topics automatically when you send messages to non-existing topics. All config options you can find … Read more
Update March 2021: With Kafka’s new KRaft mode, which entirely removes ZooKeeper from Kafka’s architecture, a Kafka cluster can handle millions of topics/partitions. See https://www.confluent.io/blog/kafka-without-zookeeper-a-sneak-peek/ for details. *short for “Kafka Raft Metadata mode”; in Early Access as of Kafka v2.8 Update September 2018: As of Kafka v2.0, a Kafka cluster can have hundreds of thousands … Read more
Kafka consumer need to commit the offset to kafka and fetch the offset from kafka. Since kafka moved the offset storage from zookeeper to kafka brokers, a kafka consumer does not need to directly communicate with zookeeper, so the new kafka consumer does not need to config the zookeeper. However, a kafka consumer always needs … Read more
In 0.11 or higher you can run the bin/kafka-delete-records.sh command to mark messages for deletion. https://github.com/apache/kafka/blob/trunk/bin/kafka-delete-records.sh For example, publish 100 messages seq 100 | ./bin/kafka-console-producer.sh –broker-list localhost:9092 –topic mytest then delete 90 of those 100 messages with the new kafka-delete-records.sh command line tool ./bin/kafka-delete-records.sh –bootstrap-server localhost:9092 –offset-json-file ./offsetfile.json where offsetfile.json contains {“partitions”: [{“topic”: “mytest”, “partition”: … Read more
1195725856 is GET[space] encoded as a big-endian, four-byte integer (see here for more information on how that works). This indicates that HTTP traffic is being sent to Kafka port 9092, but Kafka doesn’t accept HTTP traffic, it only accepts its own protocol (which takes the first four bytes as the receive size, hence the error). … Read more
When a consumer joins a consumer group it will fetch the last committed offset so it will restart to read from 5, 6, 7 if before crashing it committed the latest offset (so 4). The earliest and latest values for the auto.offset.reset property is used when a consumer starts but there is no committed offset … Read more
listeners is what the broker will use to create server sockets. advertised.listeners is what clients will use to connect to the brokers. The two settings can be different if you have a “complex” network setup (with things like public and private subnets and routing in between).
Partitioning and replication are two different things. Partitioning is for scalability. A topic is partitioned in one or more partitions distributed on different brokers so that more consumers can connect to these brokers in order to receive messages sent to the same topic but from different partitions. Increasing partitions increases scalability and the possibility to … Read more
Check for the tmp/zookeeper folder on the drive where you have kafka folder (lets say D:/), and delete the folder tmp, which will create automatically for you once run the zookeeper again.