Deriving the SparkSession
object out of SparkContext
or even SparkConf
is easy. Just that you might find the API to be slightly convoluted. Here’s an example (I’m using Spark 2.4
but this should work in the older 2.x
releases as well):
// If you already have SparkContext stored in `sc`
val spark = SparkSession.builder.config(sc.getConf).getOrCreate()
// Another example which builds a SparkConf, SparkContext and SparkSession
val conf = new SparkConf().setAppName("spark-test").setMaster("local[2]")
val sc = new SparkContext(conf)
val spark = SparkSession.builder.config(sc.getConf).getOrCreate()
Hope that helps!