How to debug Spark application locally?

Here’s how I do it using IntelliJ.

First, make sure you can run your spark application locally using spark-submit, e.g. something like:

spark-submit --class MyMainClass myapplication.jar

Then, tell your local spark driver to pause and wait for a connection from a debugger when it starts up, by adding an option like the following:

--conf spark.driver.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005

where agentlib:jdwp is the Java Debug Wire Protocol (JDWP) option, followed by a comma-separated list of sub-options:

• transport defines the connection protocol used between debugger and debuggee — either socket or “shared memory” — you almost always want socket (dt_socket) except I believe in some cases on Microsoft Windows
• server whether this process should be the server when talking to the debugger (or conversely, the client) — you always need one server and one client. In this case, we’re going to be the server and wait for a connection from the debugger
• suspend whether to pause execution until a debugger has successfully connected. We turn this on so the driver won’t start until the debugger connects
• address here, this is the port to listen on (for incoming debugger connection requests). You can set it to any available port (you just have to make sure the debugger is configured to connect to this same port)

So now, your spark-submit command line should look something like:

spark-submit \
  --name MyApp \
  --class MyMainClass \
  --conf spark.driver.extraJavaOptions=agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005

Now if you run the above, you should see something like

Listening for transport dt_socket at address: 5005

and your spark application is waiting for the debugger to attach.

Next, open the IntelliJ project containing your Spark application, and then open “Run -> Edit Configurations…” Then click the “+” to add a new run/debug configuration, and select “Remote”. Give it a name, e.g. “SparkLocal”, and select “Socket” for Transport, “Attach” for Debugger mode, and type in “localhost” for Host and the port you used above for Port, in this case, “5005”. Click “OK” to save.

In my version of IntelliJ it gives you suggestions for the debug command line to use for the debugged process, and it uses “suspend=n” — we’re ignoring that and using “suspend=y” (as above) because we want the application to wait until we connect to start.

Now you should be ready to debug. Simply start spark with the above command, then select the IntelliJ run configuration you just created and click Debug. IntelliJ should connect to your Spark application, which should now start running. You can set break points, inspect variables, etc.

Spark Shell

With spark-shell simply export SPARK_SUBMIT_OPTS as follows:

export SPARK_SUBMIT_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005

Attach to localhost:5005 using your debugger (e.g. IntelliJ IDEA) and with the Spark sources imported, you should be able to step through the code just fine.