Although it is possible to use INSERT OVERWRITE to get data out of Hive, it might not be the best method for your particular case. First let me explain what INSERT OVERWRITE does, then I’ll describe the method I use to get tsv files from Hive tables. According to the manual, your query will store … Read more
You can update a Hive partition by, for example: ALTER TABLE logs PARTITION(year = 2012, month = 12, day = 18) SET LOCATION ‘hdfs://user/darcy/logs/2012/12/18’; This command does not move the old data, nor does it delete the old data. It simply sets the partition to the new location. To drop a partition, you can do … Read more
I would say, that both of these formats have their own advantages. Parquet might be better if you have highly nested data, because it stores its elements as a tree like Google Dremel does (See here). Apache ORC might be better if your file-structure is flattened. And as far as I know parquet does not … Read more
$ hive –version Hive version 0.8.1.3 EDIT: added another ‘-‘ before the version. Doesn’t work for newer versions. Hope it works for all now. Known to work in the following distributions: HortonWorks distribution: $ hive –version Hive 0.14.0.2.2.0.0-2041 CDH 5.3 It does not work: CDH 4.3 HDinsight (Azure)
As of Hive 0.10 this patch-967 implements SHOW CREATE TABLE which “shows the CREATE TABLE statement that creates a given table, or the CREATE VIEW statement that creates a given view.” Usage: SHOW CREATE TABLE myTable;
You need to use the special hiveconf for variable substitution. e.g. hive> set CURRENT_DATE=’2012-09-16′; hive> select * from foo where day >= ${hiveconf:CURRENT_DATE} similarly, you could pass on command line: % hive -hiveconf CURRENT_DATE=’2012-09-16′ -f test.hql Note that there are env and system variables as well, so you can reference ${env:USER} for example. To see … Read more
An INNER JOIN can return data from the columns from both tables, and can duplicate values of records on either side have more than one match. A LEFT SEMI JOIN can only return columns from the left-hand table, and yields one of each record from the left-hand table where there is one or more matches … Read more
Hive has a relational database on the master node it uses to keep track of state. For instance, when you CREATE TABLE FOO(foo string) LOCATION ‘hdfs://tmp/’;, this table schema is stored in the database. If you have a partitioned table, the partitions are stored in the database(this allows hive to use lists of partitions without … Read more
Partitioning data is often used for distributing load horizontally, this has performance benefit, and helps in organizing data in a logical fashion. Example: if we are dealing with a large employee table and often run queries with WHERE clauses that restrict the results to a particular country or department . For a faster query response … Read more
MapReduce is just a computing framework. HBase has nothing to do with it. That said, you can efficiently put or fetch data to/from HBase by writing MapReduce jobs. Alternatively you can write sequential programs using other HBase APIs, such as Java, to put or fetch the data. But we use Hadoop, HBase etc to deal … Read more