Even though Kafka provides convenience methods to persist the committed offsets for a given consumer group, you’re not required to use that behavior and can write your own if you feel the need. Even so, the use of Kafka the way you’ve described it is a bit awkward for the use case as each client … Read more
Update Oct 2020: I wrote a four-part blog series on Kafka fundamentals that I’d recommend to read for questions like these. For this question in particular, take a look at part 3 on processing fundamentals. To your question: How does Kafka streaming handle concurrency? Is everything run in a single thread? I don’t see this … Read more
There are 2 ways to provide the JAAS configuration to the Kafka clients. Via the client property: sasl.jaas.config. In that case you set it to the actual JAAS configuration entry. For example, your configuration file becomes: bootstrap.servers=(address) zookeeper.connect=127.0.0.1:2181 zookeeper.connection.timeout.ms=6000 sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username=”(username)” password=”(password)”; security.protocol=SASL_PLAINTEXT sasl.mechanism=PLAIN group.id=(group) As you’ve already figured out, you can use –command-config … Read more
Here’s a Kafka Docker image which doesn’t required Zookeeper (as described above): https://hub.docker.com/r/bashj79/kafka-kraft Disclaimer: I’m the author.
This is what I’ve found so far .. Define our own custom partitioner class by implementing the kafka Partitioner interface. The implemented method will have two arguments, first the key that we provide from the producer and next the number of partition available. So we can define our own logic to set which key of … Read more
I’m surprised that all answers with “yes” are wrong. I just tested it and having the same group.id for consumers for different topic works well and does NOT mean that they share messages, because for Kafka the key is (topic, group) rather than just (group). Here is what I did: created 2 different topics T1 … Read more
The simplest way is to start a separate container inside the docker-compose file (called init-kafka in the example below) to launch the various kafka-topics –create … commands, while first making it wait for Kafka to be reachble by simply running kafka-topics –list …. Like this: version: ‘2.1’ services: zookeeper: image: confluentinc/cp-zookeeper:6.1.1 ports: – “2181:2181” environment: … Read more
Lenses Lenses (ex Landoop) enhances Kafka with User Interface, streaming SQL engine and cluster monitoring. It enables faster monitoring of Kafka data pipelines. They provide a free all-in-one docker (Lenses Box) which can serve a single broker for up to 25M messages. Note that this is recommended for development environments. Cloudera SMM Streams Messaging Manager … Read more
The reset-offsets option for kafka-consumer-groups.sh first checks to see if a consumer is active in that group before attempting to shift back your offsets. ‘Stable’ means you have an active consumer running. Use the describe-groups option to check on your consumer groups: bin/kafka-consumer-groups.sh –bootstrap-server $SERVERS –group $GROUP –describe If you see an entry under ‘CONSUMER-ID/HOST/CLIENT-ID’ … Read more
Edit bin/kafka-run-class.sh and set KAFKA_JMX_OPTS variable KAFKA_JMX_OPTS=”-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=your.kafka.broker.hostname -Djava.net.preferIPv4Stack=true” Update bin/kafka-server-start.sh add the below line export JMX_PORT=PORT