Even though Kafka provides convenience methods to persist the committed offsets for a given consumer group, you’re not required to use that behavior and can write your own if you feel the need. Even so, the use of Kafka the way you’ve described it is a bit awkward for the use case as each client … Read more
Update Oct 2020: I wrote a four-part blog series on Kafka fundamentals that I’d recommend to read for questions like these. For this question in particular, take a look at part 3 on processing fundamentals. To your question: How does Kafka streaming handle concurrency? Is everything run in a single thread? I don’t see this … Read more
Here’s a Kafka Docker image which doesn’t required Zookeeper (as described above): https://hub.docker.com/r/bashj79/kafka-kraft Disclaimer: I’m the author.
This is what I’ve found so far .. Define our own custom partitioner class by implementing the kafka Partitioner interface. The implemented method will have two arguments, first the key that we provide from the producer and next the number of partition available. So we can define our own logic to set which key of … Read more
I’m surprised that all answers with “yes” are wrong. I just tested it and having the same group.id for consumers for different topic works well and does NOT mean that they share messages, because for Kafka the key is (topic, group) rather than just (group). Here is what I did: created 2 different topics T1 … Read more
Lenses Lenses (ex Landoop) enhances Kafka with User Interface, streaming SQL engine and cluster monitoring. It enables faster monitoring of Kafka data pipelines. They provide a free all-in-one docker (Lenses Box) which can serve a single broker for up to 25M messages. Note that this is recommended for development environments. Cloudera SMM Streams Messaging Manager … Read more
The reset-offsets option for kafka-consumer-groups.sh first checks to see if a consumer is active in that group before attempting to shift back your offsets. ‘Stable’ means you have an active consumer running. Use the describe-groups option to check on your consumer groups: bin/kafka-consumer-groups.sh –bootstrap-server $SERVERS –group $GROUP –describe If you see an entry under ‘CONSUMER-ID/HOST/CLIENT-ID’ … Read more
Edit bin/kafka-run-class.sh and set KAFKA_JMX_OPTS variable KAFKA_JMX_OPTS=”-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=your.kafka.broker.hostname -Djava.net.preferIPv4Stack=true” Update bin/kafka-server-start.sh add the below line export JMX_PORT=PORT
You can see the partition size using the script /bin/kafka-log-dirs.sh /bin/kafka-log-dirs.sh –describe –bootstrap-server <KafakBrokerHost>:<KafakBrokerPort> –topic-list <YourTopic>
There are 3 possibilities: Increase request.timeout.ms – this is the time that Kafka will wait for whole batch to be ready in buffer. So in your case if there are less than 100 000 messages in buffer, timeout will occur. More info here: https://stackoverflow.com/a/34794261/2707179 Decrease batch-size – related to previous point, it will send batches … Read more