It took me a couple months of reading source code and testing things out. In a nutshell, Spark can only handle String and Binary serialization. You must manually deserialize the data. In spark, create the confluent rest service object to get the schema. Convert the schema string in the response object into an Avro schema … Read more
You can see the partition size using the script /bin/kafka-log-dirs.sh /bin/kafka-log-dirs.sh –describe –bootstrap-server <KafakBrokerHost>:<KafakBrokerPort> –topic-list <YourTopic>
There are 3 possibilities: Increase request.timeout.ms – this is the time that Kafka will wait for whole batch to be ready in buffer. So in your case if there are less than 100 000 messages in buffer, timeout will occur. More info here: https://stackoverflow.com/a/34794261/2707179 Decrease batch-size – related to previous point, it will send batches … Read more
https://cnr.sh/essays/how-paint-bike-shed-kafka-topic-naming-conventions helped us answering that same question. As a summary this article suggest to follow similar best practices to naming databases and tables, and it provides these additional points of advice: Avoid topic names based on things that change Avoid topic names based on information that would be stored in other places Avoid topic names … Read more
“metadata.broker.list” is deprecated and “bootstrap.servers” is used instead. Removing “metadata.broker.list” configuration from your code should resolve the problem. Please find more details at difference between bootstrap.servers and metadata.broker.list
According to this post, the offset is not reset: We don’t roll back offset at this moment. Since the offset is a long, it can last for a really long time. If you write 1TB a day, you can keep going for about 4 million days. Plus, you can always use more partitions (each partition … Read more
There is no access method for this. Also, before Kafka v0.10 messages do not contain any timestamp information, thus, it is impossible to know when a message was written into a topic. As of Kafka v0.10 each message contains a meta data timestamp attribute, that is either set by the producer on message creation time, … Read more
I’m not aware of any automatism, but using this simple two step approach, it should work. Note that in my case it was a partitioned topic, you can leave the params for it out in case you have a unpartitioned one: 1) Get max offset for your topic (+ their partitions): bin/kafka-run-class.sh kafka.tools.GetOffsetShell –broker-list localhost:9092 … Read more
You can adjust the JVM heap size by editing kafka-server-start.sh, zookeeper-server-start.shand so on: export KAFKA_HEAP_OPTS=”-Xmx1G -Xms1G” The -Xms parameter specifies the minimum heap size. To get your server to at least start up, try changing it to use less memory. Given that you only have 512M, you should change the maximum heap size (-Xmx) too: … Read more