In my understanding (not a big user of 2PC since I consider it limiting): Typically, 2PC is for immediate transactions. Typically, Sagas are for long running transactions. Use cases are obvious afterwards: 2PC can allow you to commit the whole transaction in a request or so, spanning this request across systems and networks. Assuming each … Read more
Apache Spark is actually built on Akka. Akka is a general purpose framework to create reactive, distributed, parallel and resilient concurrent applications in Scala or Java. Akka uses the Actor model to hide all the thread-related code and gives you really simple and helpful interfaces to implement a scalable and fault-tolerant system easily. A good … Read more
The inter_op_parallelism_threads and intra_op_parallelism_threads options are documented in the source of the tf.ConfigProto protocol buffer. These options configure two thread pools used by TensorFlow to parallelize execution, as the comments describe: // The execution of an individual op (for some op types) can be // parallelized on a pool of intra_op_parallelism_threads. // 0 means the … Read more
[*] Ah, my brain has just kicked into gear, I have a sensible suggestion now. Probably too late if this had been an interview, but never mind: Machine 1 shall be called the “control machine”, and for the sake of argument either it starts with all the data, and sends it in equal parcels to … Read more
With cache(), you use only the default storage level : MEMORY_ONLY for RDD MEMORY_AND_DISK for Dataset With persist(), you can specify which storage level you want for both RDD and Dataset. From the official docs: You can mark an RDD to be persisted using the persist() or cache() methods on it. each persisted RDD can … Read more
It is a bit more complex than you described. The auto.offset.reset config kicks in ONLY if your consumer group does not have a valid offset committed somewhere (2 supported offset storages now are Kafka and Zookeeper), and it also depends on what sort of consumer you use. If you use a high-level java consumer then … Read more
Spark uses a master/slave architecture. As you can see in the figure, it has one central coordinator (Driver) that communicates with many distributed workers (executors). The driver and each of the executors run in their own Java processes. DRIVER The driver is the process where the main method runs. First it converts the user program … Read more
It avoids a full shuffle. If it’s known that the number is decreasing then the executor can safely keep data on the minimum number of partitions, only moving the data off the extra nodes, onto the nodes that we kept. So, it would go something like this: Node 1 = 1,2,3 Node 2 = 4,5,6 … Read more
In a nutshell, ZooKeeper helps you build distributed applications. How it works You may describe ZooKeeper as a replicated synchronization service with eventual consistency. It is robust, since the persisted data is distributed between multiple nodes (this set of nodes is called an “ensemble”) and one client connects to any of them (i.e., a specific … Read more