To check if Zookeeper is accessible. One method is to simply telnet to the proper port and execute the stats command. root@host:~# telnet localhost 2181 Trying 127.0.0.1… Connected to myhost. Escape character is ‘^]’. stats Zookeeper version: 3.4.3-cdh4.0.1–1, built on 06/28/2012 23:59 GMT Clients: Latency min/avg/max: 0/0/677 Received: 4684478 Sent: 4687034 Outstanding: 0 Zxid: 0xb00187dd0 … Read more
You should also properly configure the maximum memory allocations for MapReduce. From this HortonWorks tutorial: […] Each machine in our cluster has 48 GB of RAM. Some of this RAM should be >reserved for Operating System usage. On each node, we’ll assign 40 GB RAM for >YARN to use and keep 8 GB for the … Read more
I would say, that both of these formats have their own advantages. Parquet might be better if you have highly nested data, because it stores its elements as a tree like Google Dremel does (See here). Apache ORC might be better if your file-structure is flattened. And as far as I know parquet does not … Read more
$ hive –version Hive version 0.8.1.3 EDIT: added another ‘-‘ before the version. Doesn’t work for newer versions. Hope it works for all now. Known to work in the following distributions: HortonWorks distribution: $ hive –version Hive 0.14.0.2.2.0.0-2041 CDH 5.3 It does not work: CDH 4.3 HDinsight (Azure)
Hive has a relational database on the master node it uses to keep track of state. For instance, when you CREATE TABLE FOO(foo string) LOCATION ‘hdfs://tmp/’;, this table schema is stored in the database. If you have a partitioned table, the partitions are stored in the database(this allows hive to use lists of partitions without … Read more
Prior to 0.20.203, and officially deprecated in 2.6.0: hadoop fs -dus [directory] Since 0.20.203 (dead link) 1.0.4 and still compatible through 2.6.0: hdfs dfs -du [-s] [-h] URI [URI …] You can also run hadoop fs -help for more info and specifics.
Following are the three commands which appears same but have minute differences hadoop fs {args} hadoop dfs {args} hdfs dfs {args} hadoop fs <args> FS relates to a generic file system which can point to any file systems like local, HDFS etc. So this can be used when you are dealing with different file systems … Read more
Interesting question, I spent some time looking at the code for the details and here are my thoughts. The splits are handled by the client by InputFormat.getSplits, so a look at FileInputFormat gives the following info: For each input file, get the file length, the block size and calculate the split size as max(minSize, min(maxSize, … Read more
Simple Solution: Download it from here and add to $HADOOP_HOME/bin (Source) IMPORTANT UPDATE: For hadoop-2.6.0 you can download binaries from Titus Barik blog >>. I have not only needed to point HADOOP_HOME to extracted directory [path], but also provide system property -Djava.library.path=[path]\bin to load native libs (dll).