Update Oct 2020: I wrote a four-part blog series on Kafka fundamentals that I’d recommend to read for questions like these. For this question in particular, take a look at part 3 on processing fundamentals. To your question: How does Kafka streaming handle concurrency? Is everything run in a single thread? I don’t see this … Read more
RocksDB is used for several (internal) reasons (as you mentioned already for example its performance). Conceptually, Kafka Streams does not need RocksDB — it is used as internal key-value cache and any other store offering similar functionality would work, too. Comment from @miguno below (rephrased): One important advantage of RocksDB in contrast to pure in-memory … Read more
In my case, install the missing libc6-compat didn’t work. Application still throw java.lang.UnsatisfiedLinkError. Then I find in the docker, /lib64/ld-linux-x86-64.so.2 exist and is a link to /lib/libc.musl-x86_64.so.1, but /lib only contains ld-musl-x86_64.so.1, not ld-linux-x86-64.so.2. So I add a file named ld-linux-x86-64.so.2 linked to ld-musl-x86_64.so.1 in /lib dir and solve the problem. Dockerfile I use: FROM … Read more
Update: In Kafka 2.5, a new method KStream#toTable() will be added, that will provide a convenient way to transform a KStream into a KTable. For details see: https://cwiki.apache.org/confluence/display/KAFKA/KIP-523%3A+Add+KStream%23toTable+to+the+Streams+DSL Original Answer: There is not straight forward way at the moment to do this. Your approach is absolutely valid as discussed in Confluent FAQs: http://docs.confluent.io/current/streams/faq.html#how-can-i-convert-a-kstream-to-a-ktable-without-an-aggregation-step This is … Read more
Right now, Kafka Streams offers only limited error handling capabilities. There is work in progress to simplify this. For now, your overall approach seems to be a good way to go. One comment about handling de/serialization errors: handling those error manually, requires you to do de/serialization “manually”. This means, you need to configure ByteArraySerdes for … Read more
Your question is very general, so I’ll give a general answer from my point of view. First, I’ve got two usage scenario: cases where I’m reading data from kafka, processing it and writing some output back to kafka, for these I’m using kafka streams exclusively. cases where either the data source or sink is not … Read more
Update January 2021: I wrote a four-part blog series on Kafka fundamentals that I’d recommend to read for questions like these. For this question in particular, take a look at part 3 on processing fundamentals. Update April 2018: Nowadays you can also use ksqlDB, the event streaming database for Kafka, to process your data in … Read more