apache-spark-sql – Tarik Billa

Why Presto is faster than Spark SQL [closed]

August 3, 2023 by Tarik

In general, it is hard to say if Presto is definitely faster or slower than Spark SQL. It really depends on the type of query you’re executing, environment and engine tuning parameters. However, what I see in the industry(Uber, Neflix examples) Presto is used as ad-hock SQL analytics whereas Spark for ETL/ML pipelines. One possible … Read more

Difference between df.repartition and DataFrameWriter partitionBy?

February 27, 2023 by Tarik

Watch out: I believe the accepted answer is not quite right! I’m glad you ask this question, because the behavior of these similarly-named functions differs in important and unexpected ways that are not well documented in the official spark documentation. The first part of the accepted answer is correct: calling df.repartition(COL, numPartitions=k) will create a … Read more