Spark difference between reduceByKey vs. groupByKey vs. aggregateByKey vs. combineByKey
groupByKey: Syntax: sparkContext.textFile(“hdfs://”) .flatMap(line => line.split(” “) ) .map(word => (word,1)) .groupByKey() .map((x,y) => (x,sum(y))) groupByKey can cause out of disk problems as data is sent over the network and collected on the reduced workers. reduceByKey: Syntax: sparkContext.textFile(“hdfs://”) .flatMap(line => line.split(” “)) .map(word => (word,1)) .reduceByKey((x,y)=> (x+y)) Data are combined at each partition, with only … Read more