apache-kafka – Page 7

How to change start offset for topic?

April 30, 2023 by Tarik

Since kafka 0.11.0.0 you can use the script kafka-consumer-groups.sh Example from this answer kafka-consumer-groups.sh –bootstrap-server kafka-host:9092 –group my-group –reset-offsets –to-earliest –all-topics –execute Other options listed in the KIP-122: Add Reset Consumer Group Offsets tooling .———————-.———————————————–.———————————————————————————————————————————————-. | Scenario | Arguments | Example | :———————-+———————————————–+———————————————————————————————————————————————-: | Reset to Datetime | –to-datetime YYYY-MM-DDTHH:mm:SS.sss±hh:mm | Reset to first offset … Read more

Exception in thread “main” joptsimple.UnrecognizedOptionException: zookeeper is not a recognized option

April 26, 2023 by Tarik

Newer versions(2.2+) of Kafka no longer requires ZooKeeper connection string –zookeeper localhost:2181 It throws the following exception while creating a topic Exception in thread “main” joptsimple.UnrecognizedOptionException: zookeeper is not a recognized option Instead, add Kafka Broker –bootstrap-server localhost:9092 connection string. ./kafka-topics.sh –create –topic test-topic –bootstrap-server localhost:9092 –replication-factor 1 –partitions 4

Which directory does apache kafka store the data in broker nodes

April 23, 2023 by Tarik

Kafka topics are “distributed and partitioned append only logs“. Parameter log.dir defines where topics (ie, data) is stored. It is not related to application/broker logging. The default log.dir is /tmp/kafka-logs which you may want to change in case your OS has a /tmp directory cleaner.

What is the difference between MQTT broker and Apache Kafka

April 16, 2023 by Tarik

The main motive behind Kafka is scalability. MQTT is a protocol with public specification for lightweight client / message broker communications, allowing publish/subscribe exchanges. Multiple implementations of client libraries and brokers (Mosquitto, JoramMQ…) exist and are virtually compatible. MQTT just specifies the transport, and vaguely the application part (i.e. how data is handled and possibly … Read more

Topics, partitions and keys

April 16, 2023 by Tarik

Does it mean if i want to have more than 1 consumer (from the same group) reading from one topic I need to have more than 1 partition? Let’s see the following properties of kafka: each partition is consumed by exactly one consumer in the group one consumer in the group can consume more than … Read more

What are the advantages and disadvantages of Kafka over Apache Pulsar [closed]

April 11, 2023 by Tarik

I played a bit with both lately, and here is what I gathered. Neutral: I was going to make Kafka win on the community/documentation etc. But I wasn’t able to find replies to questions I had on Kafka easily, some were old and confusing (targetting the legacy API). But Pulsar documentation is good enough, the … Read more

Delete message after consuming it in KAFKA

April 1, 2023 by Tarik

In Kafka, the responsibility of what has been consumed is the responsibility of the consumer and this is also one of the main reasons why Kafka has such great horizontal scalability. Using the high level consumer API will automatically do this for you by committing consumed offsets in Zookeeper (or a more recent configuration option … Read more

Kafka + Zookeeper: Connection to node -1 could not be established. Broker may not be available

March 30, 2023 by Tarik

UPD: if you are running in single-node mode: I have seen this message in spark console log while trying to deploy application. Solved by changing this parameter in server.properties: listeners=PLAINTEXT://myhostname:9092 to listeners=PLAINTEXT://localhost:9092 make sure that you have java process listening on 9092 with netstat -lptu

Error creating Kafka Topics- Replication factor larger than available brokers

March 24, 2023 by Tarik

Seems like your server not started , check logs for issue Kill process sudo fuser -k 2181/tcp run zookeeper bin/zookeeper-server-start.sh config/zookeeper.properties Run Kafka bin/kafka-server-start.sh config/server.properties ‘

difference between groupid and consumerid in Kafka consumer

March 21, 2023 by Tarik

Consumers groups is a Kafka abstraction that enables supporting both point-to-point and publish/subscribe messaging. A consumer can join a consumer group (let us say group_1) by setting its group.id to group_1. Consumer groups is also a way of supporting parallel consumption of the data i.e. different consumers of the same consumer group consume data in … Read more