How to know what is the reason for ClosedChannelExceptions with spark-shell in YARN client mode?

by

Reason is association with yarn cluster may be lost due to the Java 8 excessive memory allocation issue: https://issues.apache.org/jira/browse/YARN-4714

You can force YARN to ignore this by setting up the following properties in yarn-site.xml

<property>
    <name>yarn.nodemanager.pmem-check-enabled</name>
    <value>false</value>
</property>

<property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>

Thanks to simplejack,
Reference from Spark Pi Example in Cluster mode with Yarn: Association lost

Leave a Comment