You could add the path to jar file using Spark configuration at Runtime. Here is an example : conf = SparkConf().set(“spark.jars”, “/path-to-jar/spark-streaming-kafka-0-8-assembly_2.11-2.2.1.jar”) sc = SparkContext( conf=conf) Refer the document for more information.
In my environment (using docker and the image sequenceiq/spark:1.1.0-ubuntu), I ran in to this. If you look at the pyspark shell script, you’ll see that you need a few things added to your PYTHONPATH: export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.8.2.1-src.zip:$PYTHONPATH That worked in ipython for me. Update: as noted in the comments, the name of the py4j … Read more