A cleaner solution is to use standard python logging module with a custom distributed handler to collect log messages from all nodes of the spark cluster.
See “Logging in PySpark” of this Gist.
A cleaner solution is to use standard python logging module with a custom distributed handler to collect log messages from all nodes of the spark cluster.
See “Logging in PySpark” of this Gist.