Hive学习
标签（空格分隔）： Hadoop Hive
官网地址：https://hive.apache.org/
下载地址：http://www-us.apache.org/dist/hive/hive-2.1.1/
官方文档：https://cwiki.apache.org/confluence/display/Hive/GettingStarted
安装与配置
# 下载解压
wget http://www-eu.apache.org/dist/hive/hive-2.1.1/apache-hive-2.1.1-bin.tar.gz
tar zxvf apache-hive-2.1.1-bin.tar.gz
cd apache-hive-2.1.1-bin
# 配置mysql
cp conf/hive-default.xml.template conf/hive-site.xml
mv /path/to/mysql-connector-java-5.1.23-bin.jar lib/
vim conf/hive-site.xml
修改以下属性：
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://127.0.0.1:3306/hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>password</value>
</property>
配置全局变量：
- 将hive的bin目录加入PATH中，方便运行
运行：
# run the schematool command below as an initialization step
schematool --dbType mysql -initSchema
# To run HiveServer2 and Beeline from shell:
hiveserver2
beeline -u jdbc:hive2://localhost:10000
# Or to start Beeline and HiveServer2 in the same process for testing purpose
beeline -u jdbc:hive2://
异常：
Relative path in absolute URI: ${system:user.name%7D
Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D
解决方法：
在hive-site.xml中加入：
<property>
<name>system:java.io.tmpdir</name>
<value>/home/lyndon/tmp/hive</value>
</property>
<property>
<name>system:user.name</name>
<value>lyndon</value>
</property>
安装Hive的web管理界面
wget http://www-us.apache.org/dist/hive/hive-2.1.1/apache-hive-2.1.1-src.tar.gz
tar zxvf apache-hive-2.1.1-src.tar.gz
cd apache-hive-2.1.1-src/hwi
jar cvfM0 hive-hwi-2.1.1.war -C web/ .
cp hive-hwi-2.1.1.war /opt/hive/lib
https://cwiki.apache.org/confluence/display/Hive/HiveWebInterface
HWI is only available in Hive releases prior to 2.2.0. It was removed by HIVE-15622.
配置失败
Hive数据导入
使用Load
- 语法：
LOAD DATA [LOCAL(是否为本地文件/HDFS上)] INPATH ‘filepath’ [OVERWRITE]
INTO TABLE tablename [PARTITION (partcol1=val1, partcol2=val2…)]
使用Sqoop
- 使用1.99.7版本
下载地址： http://www-eu.apache.org/dist/sqoop/1.99.7/
官方文档： http://sqoop.apache.org/docs/1.99.7/
axel -n 10 http://www-eu.apache.org/dist/sqoop/1.99.7/sqoop-1.99.7-bin-hadoop200.tar.gz
tar zxvf sqoop-1.99.7-bin-hadoop200.tar.gz
sudo mv sqoop-1.99.7-bin-hadoop200 /usr/lib/sqoop
sudo mkdir -p /var/lib/lyndon
sudo cp ~/Downloads/mysql-connector-java-5.1.23-bin.jar /var/lib/lyndon/
export SQOOP_SERVER_EXTRA_LIB=/var/lib/lyndon/
...未继续
- 使用1.4.6版本
查看分隔符：
desc formatted stations;默认:\1(SOH)
修改HIVE序列化存储分隔符：alter table tablename set SERDEPROPERTIES('field.delim'=',');
- 安装与配置
axel -n 10 http://www-us.apache.org/dist/sqoop/1.4.6/sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
tar zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
sudo tar zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
cd sqoop-1.4.6.bin__hadoop-2.0.4-alpha
cd bin
sudo cp ~/Downloads/mysql-connector-java-5.1.23-bin.jar ../lib
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
导入到HDFS：
./sqoop import --connect jdbc:mysql://127.0.0.1:3306/test --username root --password l --table stations --target-dir '/sqoop/stations'导入到HIVE：
./sqoop import --hive-import --connect jdbc:mysql://127.0.0.1:3306/test --username root --password l --table stations自定义表名，添加筛选条件：
./sqoop import --hive-import --connect jdbc:mysql://127.0.0.1:3306/test --username root --password l --table stations --hive-table stations_bj --where 'fchar like "%bj%"'使用query查询语句:
使用–query，不能包含–table参数，必须包含–target-dir和–split-by
条件中必须有 $CONDITIONS
./sqoop import --hive-import --connect jdbc:mysql://127.0.0.1:3306/test --username root --password l --hive-table stations_q --query 'select * from stations where fchar="bj" and $CONDITIONS' --target-dir '/sqoop/stations_q' --split-by 'id'导出到MYSQL：
必须先在MYSQL中创建此表，且对应列关系
./sqoop export --connect jdbc:mysql://127.0.0.1:3306/test --username root --password l -m 1 --table stations_bj --export-dir '/user/hive/warehouse/stations_q'
JDBC接口
注意
- 需要导入Hive中lib下面的jar包
- 注册JDBC驱动使用
org.apache.hive.jdbc.HiveDriver - 创建链接的URL为：
jdbc:hive2://localhost:10000需要写入用户名，否则报权限异常。如：Connection connection = DriverManager.getConnection("jdbc:hive2://localhost:10000", "lyndon", ""); - 使用
statement.execute()执行不返回ResultSet的sql语句，否则报错。
示例代码：
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;
/**
* Created by lyndon on 17-4-2.
*/
public class HiveJdbcClient {
public static void main(String[] args) throws Exception {
// 注册JDBC驱动
try {
Class.forName("org.apache.hive.jdbc.HiveDriver");
} catch (ClassNotFoundException e) {
e.printStackTrace();
System.exit(1);
}
// 创建链接
Connection connection = DriverManager.getConnection("jdbc:hive2://localhost:10000", "lyndon", "");
Statement statement = connection.createStatement();
String tableName = "u1_data";
statement.execute("DROP TABLE " + tableName);
statement.execute("CREATE TABLE " + tableName +
"(userid int," +
"movieid int," +
"rating int," +
"unixtime int)" +
"row format delimited fields terminated by ','" +
"stored as textfile");
String sql = "show tables";
System.out.println("Running: " + sql);
ResultSet resultSet = statement.executeQuery(sql);
while (resultSet.next()) {
System.out.println(resultSet.getString(1));
}
// load data
String filepath = "/home/lyndon/install/ratings.csv";
sql = "load data local inpath '" + filepath + "' overwrite into table " + tableName;
System.out.println("Running: " + sql);
statement.execute(sql);
// 查询
sql = "SELECT * FROM " + tableName + " LIMIT 5";
System.out.println("Running: " + sql);
resultSet = statement.executeQuery(sql);
while (resultSet.next()) {
System.out.println(resultSet.getString(2)+"\t"+resultSet.getString(3));
}
}
}
