In order to keep everything on the grid use hadoop streaming with a single reducer and cat as the mapper and reducer (basically a noop) – add compression using MR flags. hadoop jar \ $HADOOP_PREFIX/share/hadoop/tools/lib/hadoop-streaming.jar \<br> -Dmapred.reduce.tasks=1 \ -Dmapred.job.queue.name=$QUEUE \ -input “$INPUT” \ -output “$OUTPUT” \ -mapper cat \ -reducer cat If you want compression … Read more
There is no surety if pig uses the configuration DEFAULT_PARALLEL value for every steps in the pig script. Try PARALLEL along with your specific join/group step which you feel taking time (In your case GROUP step). inputDataGrouped = GROUP inputData BY (group_name) PARALLEL 67; If still it is not working then you might have to … Read more
You won’t be able “open” the file using a hdfs dfs -text because its not a text file. Parquet files are written to disk very differently compared to text files. And for the same matter, the Parquet project provides parquet-tools to do tasks like which you are trying to do. Open and see the schema, … Read more
COUNT is part of pig see the manual LOGS= LOAD ‘log’; LOGS_GROUP= GROUP LOGS ALL; LOG_COUNT = FOREACH LOGS_GROUP GENERATE COUNT(LOGS);
MapReduce is just a computing framework. HBase has nothing to do with it. That said, you can efficiently put or fetch data to/from HBase by writing MapReduce jobs. Alternatively you can write sequential programs using other HBase APIs, such as Java, to put or fetch the data. But we use Hadoop, HBase etc to deal … Read more
Check out this post from Alan Gates, Pig architect at Yahoo!, that compares when would use a SQL like Hive rather than Pig. He makes a very convincing case as to the usefulness of a procedural language like Pig (vs. declarative SQL) and its utility to dataflow designers.