kafka-python – Tarik Billa

How to get latest offset for a partition for a kafka topic?

September 5, 2023 by Tarik

Finally after spending a day on this and several false starts, I was able to find a solution and get it working. Posting it her so that others may refer to it. from kafka import SimpleClient from kafka.protocol.offset import OffsetRequest, OffsetResetStrategy from kafka.common import OffsetRequestPayload client = SimpleClient(brokers) partitions = client.topic_partitions[topic] offset_requests = [OffsetRequestPayload(topic, p, … Read more

SyntaxError on “self.async” when running python kafka producer

July 29, 2023 by Tarik

You need to install kafka-python, not kafka package. If kafka is installed, then you need to pip uninstall kafka and replace with the other https://github.com/dpkp/kafka-python/issues/1566

How to programmatically create a topic in Apache Kafka using Python

May 2, 2023 by Tarik

You can programmatically create topics using either kafka-python or confluent_kafka client which is a lightweight wrapper around librdkafka. Using kafka-python from kafka.admin import KafkaAdminClient, NewTopic admin_client = KafkaAdminClient( bootstrap_servers=”localhost:9092″, client_id=’test’ ) topic_list = [] topic_list.append(NewTopic(name=”example_topic”, num_partitions=1, replication_factor=1)) admin_client.create_topics(new_topics=topic_list, validate_only=False) Using confluent_kafka from confluent_kafka.admin import AdminClient, NewTopic admin_client = AdminClient({ “bootstrap.servers”: “localhost:9092” }) topic_list = [] … Read more

Kafka in Docker not working

February 24, 2023 by Tarik

My solution to this issue is slightly different. I configure Kafka to advertise on kafka host and, because it’s exposed on the host machine on localhost:9092, I add an entry in /etc/hosts for kafka to resolve to localhost. By doing this Kafka can be accessed from both other Docker containers and from localhost. docker-compose.yml: my-web-service: … Read more