apache-spark – Page 20

How to stop INFO messages displaying on spark console?

October 13, 2022 by Tarik

Edit your conf/log4j.properties file and change the following line: log4j.rootCategory=INFO, console to log4j.rootCategory=ERROR, console Another approach would be to : Start spark-shell and type in the following: import org.apache.log4j.Logger import org.apache.log4j.Level Logger.getLogger(“org”).setLevel(Level.OFF) Logger.getLogger(“akka”).setLevel(Level.OFF) You won’t see any logs after that. Other options for Level include: all, debug, error, fatal, info, off, trace, trace_int, warn Details … Read more

What is the difference between cache and persist?

October 10, 2022 by Tarik

With cache(), you use only the default storage level : MEMORY_ONLY for RDD MEMORY_AND_DISK for Dataset With persist(), you can specify which storage level you want for both RDD and Dataset. From the official docs: You can mark an RDD to be persisted using the persist() or cache() methods on it. each persisted RDD can … Read more

What are workers, executors, cores in Spark Standalone cluster?

October 3, 2022 by Tarik

Spark uses a master/slave architecture. As you can see in the figure, it has one central coordinator (Driver) that communicates with many distributed workers (executors). The driver and each of the executors run in their own Java processes. DRIVER The driver is the process where the main method runs. First it converts the user program … Read more

How to show full column content in a Spark Dataframe?

September 30, 2022 by Tarik

results.show(20, false) will not truncate. Check the source 20 is the default number of rows displayed when show() is called without any arguments.

What is the difference between map and flatMap and a good use case for each?

September 30, 2022 by Tarik

Here is an example of the difference, as a spark-shell session: First, some data – two lines of text: val rdd = sc.parallelize(Seq(“Roses are red”, “Violets are blue”)) // lines rdd.collect res0: Array[String] = Array(“Roses are red”, “Violets are blue”) Now, map transforms an RDD of length N into another RDD of length N. For … Read more

Spark – repartition() vs coalesce()

September 23, 2022 by Tarik

It avoids a full shuffle. If it’s known that the number is decreasing then the executor can safely keep data on the minimum number of partitions, only moving the data off the extra nodes, onto the nodes that we kept. So, it would go something like this: Node 1 = 1,2,3 Node 2 = 4,5,6 … Read more