I was getting the same error while creating Data frames on Spark Shell :
Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /metastore_db.
Cause:
I found that this is happening as there were multiple other instances of Spark-Shell already running and holding derby DB already, so when i was starting yet another Spark Shell and creating Data Frame on it using RDD.toDF() it was throwing error:
Solution:
I ran the ps command to find other instances of Spark-Shell:
ps -ef | grep spark-shell
and i killed them all using kill command:
kill -9 Spark-Shell-processID ( example: kill -9 4848)
after all the SPark-Shell instances were gone, i started a new SPark SHell and reran my Data frame function and it ran just fine 🙂