Yes, Paxos provides guarantees that are not provided by the Dynamo-like systems and their read-write quorums. The difference is how failures are handled and what happens during a write. After a successful write, both kind of systems behave similarly. The data will be saved and available for reading afterwards (until overwritten or deleted) and so … Read more
First of all, zookeeper is needed only for high level consumer. SimpleConsumer does not require zookeeper to work. The main reason zookeeper is needed for a high level consumer is to track consumed offsets and handle load balancing. Now in more detail. Regarding offset tracking, imagine following scenario: you start a consumer, consume 100 messages … Read more
Check for the tmp/zookeeper folder on the drive where you have kafka folder (lets say D:/), and delete the folder tmp, which will create automatically for you once run the zookeeper again.
Netfix opensourced Curator a framework to make use of Zookeeper even more convenient. It has build in test server class. Just add this test dependency to your project descriptor be it maven, gradle or else: org.apache.curator:curator-framework:4.0.1 org.apache.curator:curator-test:4.0.1 And here are the test essentials. TestingServer zkTestServer; CuratorFramework cli; @Before public void startZookeeper() throws Exception { zkTestServer … Read more
As of v0.9.0, Kafka ships with a suite of tools in the /bin one of which is the kafka-consumer-groups.sh tool. This will delete a consumer group. ./kafka-consumer-groups.sh –zookeeper <zookeeper_url> –delete –group <group-name>
I am going to implement the orchestration of a set of microservices in my application. This is a hard problem to solve by yourself. You are probably better off using an existing orchestration system (see below). Is there any other powerful tool? You should look into kubernetes, which seems to be the standard in orchestration … Read more
I’m asking myself the same question: Where to put my docker-compose.yml file? I’ve decided to do it the following way: One Repo for the docker-compose.yml along with deployment scripts (Jenkins pipeline in my case). One Repo per micro service along with a Dockerfile and its build logic (built image is pushed to private docker registry). … Read more
Edit – Zookeeper is not required in newer version of Kafka. Please see answer by @Neeleshkumar Srinivasan Mannur for API version 0.11.0+ Original answer I fixed it.. After a long research.. ZkClient zkClient = new ZkClient(“localhost:2181”, 10000, 10000); AdminUtils.createTopic(zkClient, myTopic, 10, 1, new Properties()); From the above code, ZkClient will create a topic but this … Read more
Older versions of Kafka (pre 0.9) store offsets in ZK only, while newer version of Kafka, by default store offsets in an internal Kafka topic called __consumer_offsets (newer version might still commit to ZK though). The advantage of committing offsets to the broker is, that the consumer does not depend on ZK and thus clients … Read more