overwriting a spark output using pyspark

Try:

spark_df.write.format('com.databricks.spark.csv') \
  .mode('overwrite').option("header", "true").save(self.output_file_path)