Aggregating multiple columns with custom function in Spark
Consider using the struct function to group the columns together before collecting as a list: import org.apache.spark.sql.functions.{collect_list, struct} import sqlContext.implicits._ val df = Seq( (“john”, “tomato”, 1.99), (“john”, “carrot”, 0.45), (“bill”, “apple”, 0.99), (“john”, “banana”, 1.29), (“bill”, “taco”, 2.59) ).toDF(“name”, “food”, “price”) df.groupBy($”name”) .agg(collect_list(struct($”food”, $”price”)).as(“foods”)) .show(false) Outputs: +—-+———————————————+ |name|foods | +—-+———————————————+ |john|[[tomato,1.99], [carrot,0.45], [banana,1.29]]| |bill|[[apple,0.99], … Read more