Kafka is a fast, scalable, distributed in nature by its design, partitioned and replicated commit log service.So there is no priority on topic or message. I also faced same problem that you have.Solution is very simple.Create topics in kafka queue,Let say: high_priority_queue medium_priority_queue low_priority_queue Publish high priority message in high_priority_queue and medium priority message in … Read more
Apache Kafka doesn’t support decreasing the partition number. You should see the topic as a whole and the partitions are a way for scaling out improving performance. So all data sent to topic flow to all partitions and removing one of them means data loss.
You can use the excellent kafkacat tool. Sample command: kafkacat -b kafka-broker:9092 -t my_topic_name -C \ -f ‘\nKey (%K bytes): %k Value (%S bytes): %s Timestamp: %T Partition: %p Offset: %o Headers: %h\n’ Sample output: Key (-1 bytes): Value (13 bytes): {foo:”bar 5″} Timestamp: 1548350164096 Partition: 0 Offset: 34 Headers: __connect.errors.topic=test_topic_json,__connect.errors.partition=0,__connect.errors.offset=94,__connect.errors.connector.name=file_sink_03,__connect.errors.task.id=0,__connect.errors.stage=VALU E_CONVERTER,__connect.errors.class.name=org.apache.kafka.connect.json.JsonConverter,__connect.errors.exception.class.name=org.apache.kafka.connect.errors.DataException,__connect.errors.exception.message=Co nverting byte[] to … Read more
Looking at the documentation you linked to it looks like the main difference between the Future and the Callback lies in who initiates the “request is finished, what now?” question. Let’s say we have a customer C and a baker B. And C is asking B to make him a nice cookie. Now there are … Read more
I was having this same problem today. I’m a newbie at Kafka and was simply trying to get a sample Java producer and consumer running. I was able to get the consumer working, but kept getting the same “topic not present in metadata” error as you, with the producer. Finally, out of desperation, I added … Read more
kafka provides publish/subscribe messaging as a distributed commit log. Usually you install kafka on each host where you need to produce some data to be forwarded somewhere else and all those hosts will together form a cluster. The good thing here is that if for some reason network connectivity becomes unstable or goes down, your … Read more
Fields cannot be renamed in BACKWARD compatibility mode. As a workaround you can change the compatibility rules for the schema registry. According to the docs: The schema registry server can enforce certain compatibility rules when new schemas are registered in a subject. Currently, we support the following compatibility rules. Backward compatibility (default): A new schema … Read more
If you’re running Kafka in Docker (or similar) you need to configure the listeners correctly. This article describes it in detail. Here’s an example of a Docker Compose that you can use to access Kafka from your host machine. Disclaimer: I wrote the article 🙂
We have solved this in 1.7 by changing the headless service to Type=NodePort and setting the externalTrafficPolicy=Local. This bypasses the internal load balancing of a Service and traffic destined to a specific node on that node port will only work if a Kafka pod is on that node. apiVersion: v1 kind: Service metadata: name: broker … Read more
Kafka v0.11.0.0 adds support for custom headers. You can add them when creating a ProducerRecord like this: new ProducerRecord(key, value, headers, …), where headers is of type Iterable<Header> For more details see: https://issues.apache.org/jira/browse/KAFKA-4208 https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers