Find port number where HDFS is listening

Below command available in Apache hadoop 2.7.0 onwards, this can be used for getting the values for the hadoop configuration properties. fs.default.name is deprecated in hadoop 2.0, fs.defaultFS is the updated value. Not sure whether this will work incase of maprfs. hdfs getconf -confKey fs.defaultFS # ( new property ) or hdfs getconf -confKey fs.default.name … Read more

Where are logs in Spark on YARN?

You can access logs through the command yarn logs -applicationId <application ID> [OPTIONS] general options are: appOwner <Application Owner> – AppOwner (assumed to be current user if not specified) containerId <Container ID> – ContainerId (must be specified if node address is specified) nodeAddress <Node Address> – NodeAddress in the format nodename:port (must be specified if … Read more

Buiding Hadoop with Eclipse / Maven – Missing artifact jdk.tools:jdk.tools:jar:1.6

The problem is in the Eclipse Maven support, the related question is here. Under Eclipse, the java.home variable is set to the JRE that was used to start Eclipse, not the build JRE. The default system JRE from C:\Program Files doesn’t include the JDK so tools.jar is not being found. To fix the issue you … Read more