Hive学习

2023年12月15日 219次阅读 来源: 予早

Hive学习

标签（空格分隔）： Hadoop Hive

官网地址：https://hive.apache.org/
下载地址：http://www-us.apache.org/dist/hive/hive-2.1.1/
官方文档：https://cwiki.apache.org/confluence/display/Hive/GettingStarted

安装与配置

# 下载解压
wget http://www-eu.apache.org/dist/hive/hive-2.1.1/apache-hive-2.1.1-bin.tar.gz
tar zxvf apache-hive-2.1.1-bin.tar.gz
cd apache-hive-2.1.1-bin
# 配置mysql
cp conf/hive-default.xml.template conf/hive-site.xml
mv /path/to/mysql-connector-java-5.1.23-bin.jar lib/
vim conf/hive-site.xml

修改以下属性：

<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://127.0.0.1:3306/hive</value>
</property>
<property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
</property>
<property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>root</value>
</property>
<property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>password</value>
</property>

配置全局变量：

  • 将hive的bin目录加入PATH中，方便运行

运行：

# run the schematool command below as an initialization step
schematool --dbType mysql -initSchema
# To run HiveServer2 and Beeline from shell:
hiveserver2
beeline -u jdbc:hive2://localhost:10000

# Or to start Beeline and HiveServer2 in the same process for testing purpose
beeline -u jdbc:hive2://

异常：
Relative path in absolute URI: ${system:user.name%7D
Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D

解决方法：
在hive-site.xml中加入：

<property> 
  <name>system:java.io.tmpdir</name> 
  <value>/home/lyndon/tmp/hive</value> 
</property> 
<property> 
  <name>system:user.name</name> 
  <value>lyndon</value> 
</property>

安装Hive的web管理界面

wget http://www-us.apache.org/dist/hive/hive-2.1.1/apache-hive-2.1.1-src.tar.gz
tar zxvf apache-hive-2.1.1-src.tar.gz
cd apache-hive-2.1.1-src/hwi
jar cvfM0 hive-hwi-2.1.1.war -C web/ .
cp hive-hwi-2.1.1.war /opt/hive/lib

https://cwiki.apache.org/confluence/display/Hive/HiveWebInterface
HWI is only available in Hive releases prior to 2.2.0. It was removed by HIVE-15622.

配置失败

Hive数据导入

使用Load

  • 语法：
    LOAD DATA [LOCAL(是否为本地文件/HDFS上)] INPATH ‘filepath’ [OVERWRITE]
    INTO TABLE tablename [PARTITION (partcol1=val1, partcol2=val2…)]

使用Sqoop

官网：http://sqoop.apache.org

  • 使用1.99.7版本

下载地址： http://www-eu.apache.org/dist/sqoop/1.99.7/
官方文档： http://sqoop.apache.org/docs/1.99.7/

axel -n 10 http://www-eu.apache.org/dist/sqoop/1.99.7/sqoop-1.99.7-bin-hadoop200.tar.gz
tar zxvf sqoop-1.99.7-bin-hadoop200.tar.gz
sudo mv sqoop-1.99.7-bin-hadoop200 /usr/lib/sqoop
sudo mkdir -p /var/lib/lyndon
sudo cp ~/Downloads/mysql-connector-java-5.1.23-bin.jar /var/lib/lyndon/
export SQOOP_SERVER_EXTRA_LIB=/var/lib/lyndon/
...未继续
  • 使用1.4.6版本

查看分隔符：desc formatted stations; 默认:\1(SOH)
修改HIVE序列化存储分隔符： alter table tablename set SERDEPROPERTIES('field.delim'=',');

  1. 安装与配置
axel -n 10 http://www-us.apache.org/dist/sqoop/1.4.6/sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
tar zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
sudo tar zxvf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
cd sqoop-1.4.6.bin__hadoop-2.0.4-alpha
cd bin
sudo cp ~/Downloads/mysql-connector-java-5.1.23-bin.jar ../lib
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME

  1. 导入到HDFS：
    ./sqoop import --connect jdbc:mysql://127.0.0.1:3306/test --username root --password l --table stations --target-dir '/sqoop/stations'

  2. 导入到HIVE：
    ./sqoop import --hive-import --connect jdbc:mysql://127.0.0.1:3306/test --username root --password l --table stations

  3. 自定义表名，添加筛选条件：
    ./sqoop import --hive-import --connect jdbc:mysql://127.0.0.1:3306/test --username root --password l --table stations --hive-table stations_bj --where 'fchar like "%bj%"'

  4. 使用query查询语句:
    使用–query，不能包含–table参数，必须包含–target-dir和–split-by
    条件中必须有 $CONDITIONS
    ./sqoop import --hive-import --connect jdbc:mysql://127.0.0.1:3306/test --username root --password l --hive-table stations_q --query 'select * from stations where fchar="bj" and $CONDITIONS' --target-dir '/sqoop/stations_q' --split-by 'id'

  5. 导出到MYSQL：
    必须先在MYSQL中创建此表，且对应列关系
    ./sqoop export --connect jdbc:mysql://127.0.0.1:3306/test --username root --password l -m 1 --table stations_bj --export-dir '/user/hive/warehouse/stations_q'

JDBC接口

注意

  • 需要导入Hive中lib下面的jar包
  • 注册JDBC驱动使用org.apache.hive.jdbc.HiveDriver
  • 创建链接的URL为：jdbc:hive2://localhost:10000 需要写入用户名，否则报权限异常。如：Connection connection = DriverManager.getConnection("jdbc:hive2://localhost:10000", "lyndon", "");
  • 使用statement.execute()执行不返回ResultSet的sql语句，否则报错。

示例代码：

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;

/**
 * Created by lyndon on 17-4-2.
 */
public class HiveJdbcClient {
    public static void main(String[] args) throws Exception {
//        注册JDBC驱动
        try {
            Class.forName("org.apache.hive.jdbc.HiveDriver");
        } catch (ClassNotFoundException e) {
            e.printStackTrace();
            System.exit(1);
        }

//        创建链接
        Connection connection = DriverManager.getConnection("jdbc:hive2://localhost:10000", "lyndon", "");
        Statement statement = connection.createStatement();

        String tableName = "u1_data";
        statement.execute("DROP TABLE " + tableName);
        statement.execute("CREATE TABLE " + tableName +
                "(userid int," +
                "movieid int," +
                "rating int," +
                "unixtime int)" +
                "row format delimited fields terminated by ','" +
                "stored as textfile");
        String sql = "show tables";
        System.out.println("Running: " + sql);
        ResultSet resultSet = statement.executeQuery(sql);
        while (resultSet.next()) {
            System.out.println(resultSet.getString(1));
        }

//        load data
        String filepath = "/home/lyndon/install/ratings.csv";
        sql = "load data local inpath '" + filepath + "' overwrite into table " + tableName;
        System.out.println("Running: " + sql);
        statement.execute(sql);

//        查询
        sql = "SELECT * FROM " + tableName + " LIMIT 5";
        System.out.println("Running: " + sql);
        resultSet = statement.executeQuery(sql);
        while (resultSet.next()) {
            System.out.println(resultSet.getString(2)+"\t"+resultSet.getString(3));
        }

    }
}
    原文作者：予早
    原文地址: https://www.jianshu.com/p/ed3dcc1c3ca4
    本文转自网络文章，转载此文章仅为分享知识，如有侵权，请联系博主进行删除。
点赞