Prior to 0.20.203, and officially deprecated in 2.6.0: hadoop fs -dus [directory] Since 0.20.203 (dead link) 1.0.4 and still compatible through 2.6.0: hdfs dfs -du [-s] [-h] URI [URI …] You can also run hadoop fs -help for more info and specifics.
Following are the three commands which appears same but have minute differences hadoop fs {args} hadoop dfs {args} hdfs dfs {args} hadoop fs <args> FS relates to a generic file system which can point to any file systems like local, HDFS etc. So this can be used when you are dealing with different file systems … Read more
If install Hadoop on Mac OSX, make sure turn on Remote Login under System Preferences then File Sharing. This worked on my machine.
Interesting question, I spent some time looking at the code for the details and here are my thoughts. The splits are handled by the client by InputFormat.getSplits, so a look at FileInputFormat gives the following info: For each input file, get the file length, the block size and calculate the split size as max(minSize, min(maxSize, … Read more
Simple Solution: Download it from here and add to $HADOOP_HOME/bin (Source) IMPORTANT UPDATE: For hadoop-2.6.0 you can download binaries from Titus Barik blog >>. I have not only needed to point HADOOP_HOME to extracted directory [path], but also provide system property -Djava.library.path=[path]\bin to load native libs (dll).
Avro is a Row based format. If you want to retrieve the data as a whole you can use Avro Parquet is a Column based format. If your data consists of a lot of columns but you are interested in a subset of columns then you can use Parquet HBase is useful when frequent updating … Read more
I think this tutorial on Yahoo’s developer network will help you with this: Chaining Jobs You use the JobClient.runJob(). The output path of the data from the first job becomes the input path to your second job. These need to be passed in as arguments to your jobs with appropriate code to parse them and … Read more
In order to forcefully let the namenode leave safemode, following command should be executed: bin/hadoop dfsadmin -safemode leave You are getting Unknown command error for your command as -safemode isn’t a sub-command for hadoop fs, but it is of hadoop dfsadmin. Also after the above command, I would suggest you to once run hadoop fsck … Read more
First of all shuffling is the process of transfering data from the mappers to the reducers, so I think it is obvious that it is necessary for the reducers, since otherwise, they wouldn’t be able to have any input (or input from every mapper). Shuffling can start even before the map phase has finished, to … Read more
Hadoop is basically 3 things, a FS (Hadoop Distributed File System), a computation framework (MapReduce) and a management bridge (Yet Another Resource Negotiator). HDFS allows you store huge amounts of data in a distributed (provides faster read/write access) and redundant (provides better availability) manner. And MapReduce allows you to process this huge data in a … Read more