apache-kafka

Is it possible to add partitions to an existing topic in Kafka 0.8.2

by

Looks like you can use this script instead: bin/kafka-topics.sh –zookeeper zk_host:port/chroot –alter –topic my_topic_name –partitions 40 In the code it looks like they do same thing: AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK(topic, partitionReplicaList, zkClient, true) kafka-topics.sh executes this piece of code as well as AddPartitionsCommand used by kafka-add-partition script. However you have to be aware of re-partitioning when using key: … Read more

Delete topic in Kafka 0.8.1.1

by

Deleting topic isn’t always working in 0.8.1.1 Deletion should be working in the next release, 0.8.2 kafka-topics.sh –delete –zookeeper localhost:2181 –topic your_topic_name Topic your_topic_name is marked for deletion. Note: This will have no impact if delete.topic.enable is not set to true. You may also pass in a bootstrap server instead of zookeeper: kafka-topics.sh –bootstrap-server kafka:9092 … Read more

What’s the purpose of Kafka’s key/value pair-based messaging?

by

Kafka uses the abstraction of a distributed log that consists of partitions. Splitting a log into partitions allows to scale-out the system. Keys are used to determine the partition within a log to which a message get’s appended to. While the value is the actual payload of the message. The examples are actually not very … Read more

Kafka: Consumer API vs Streams API

by

Update January 2021: I wrote a four-part blog series on Kafka fundamentals that I’d recommend to read for questions like these. For this question in particular, take a look at part 3 on processing fundamentals. Update April 2018: Nowadays you can also use ksqlDB, the event streaming database for Kafka, to process your data in … Read more

How can I send large messages with Kafka (over 15MB)?

by

You need to adjust three (or four) properties: Consumer side:fetch.message.max.bytes – this will determine the largest size of a message that can be fetched by the consumer. Broker side: replica.fetch.max.bytes – this will allow for the replicas in the brokers to send messages within the cluster and make sure the messages are replicated correctly. If … Read more

Is key required as part of sending messages to Kafka?

by

Keys are mostly useful/necessary if you require strong order for a key and are developing something like a state machine. If you require that messages with the same key (for instance, a unique id) are always seen in the correct order, attaching a key to messages will ensure messages with the same key always go … Read more