How to change job/stage description in web UI?

That’s where one of the very uncommon features of Spark Core called local properties applies so well.

Spark SQL uses it to group different Spark jobs under a single structured query so you can use SQL tab and navigate easily.

You can control local properties using SparkContext.setLocalProperty:

Set a local property that affects jobs submitted from this thread, such as the Spark fair scheduler pool. User-defined properties may also be set here. These properties are propagated through to worker tasks and can be accessed there via org.apache.spark.TaskContext#getLocalProperty.

web UI uses two local properties:

• callSite.short in Jobs tab (and is exactly what you want)
• callSite.long in Job Details page.

Sample Usage

scala> sc.setLocalProperty("callSite.short", "callSite.short")

scala> sc.setLocalProperty("callSite.long", "this is callSite.long")

scala> sc.parallelize(0 to 9).count
res2: Long = 10

And the result in web UI.

Click a job to see the details where you can find the longer call site, i.e. callSite.long.

Here comes the Stages tab.