首先确保Hadoop环境和Hive环境兼容且配置好,我这里Hive的元数据库使用的MySQL。
编辑$HIVE_HOME/conf/hive-site.xml,对server2适当配置并启动server。
使用server2是因为HiveServer2支持多客户端的并发和认证,为开放API客户端如JDBC、ODBC提供更好的支持。server/server2区别以及配置
当然也可以不配置server2相关内容,保留默认配置即可,需注意默认tcp port is 10000(default http port is 10001)
启动server2:hive --service hiveserver2 &
使用beeline连接,在没有配置用户的情况下任何人都可以连接,密码为空:
$ beeline Beeline version 1.2.1.spark2 by Apache Hive beeline> !connect jdbc:hive2://localhost:10000 hadoop Connecting to jdbc:hive2://localhost:10000 Enter password for jdbc:hive2://localhost:10000: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/lib/spark-2.2.0/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/lib/hadoop-2.8.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Connected to: Apache Hive (version 2.3.0) Driver: Hive JDBC (version 1.2.1.spark2) Transaction isolation: TRANSACTION_REPEATABLE_READ
编码完成
CustomHiveServer2Auth
类,以及Authentication file
authentication file是保存明文用户名和MD5加密密码的文本文件,如下所示。该文件需要放置在server中,我这里放在$HIVE_HOME/conf下并命名为hive.server2.users.conf了。opticalix,88b6ed7f53340e0d45ff60f277e4c420 other_username,88b6ed7f53340e0d45ff60f277e4c421
CustomHiveServer2Auth
类实现如下,参考博文见此:package com.opticalix; import com.opticalix.utils.MD5Utils; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hive.conf.HiveConf; import org.apache.hive.service.auth.PasswdAuthenticationProvider; import javax.security.sasl.AuthenticationException; import java.io.BufferedReader; import java.io.File; import java.io.FileReader; import java.io.IOException; /** * 根据用户名密码文件,遍历看是否能匹配上一个用户。如果不能则抛出异常 */ public class CustomHiveServer2Auth implements PasswdAuthenticationProvider { public static final String HIVE_SERVER2_CUSTOM_AUTHENTICATION_FILE = "hive.server2.custom.authentication.file"; @Override public void Authenticate(String username, String password) throws AuthenticationException { boolean ok = false; String passMd5 = new MD5Utils().md5(password); //load hive conf, get auth file location HiveConf hiveConf = new HiveConf(); Configuration conf = new Configuration(hiveConf); String filePath = conf.get(HIVE_SERVER2_CUSTOM_AUTHENTICATION_FILE); System.out.println("hive.server2.custom.authentication.file [" + filePath + "] .."); //try match File file = new File(filePath); BufferedReader reader = null; try { reader = new BufferedReader(new FileReader(file)); String tempString = null; while ((tempString = reader.readLine()) != null) { System.out.println("read line=" + tempString); String[] split = tempString.split(",", -1); if (split.length != 2) continue; //ok if (split[0].equals(username) && split[1].equals(passMd5)) { ok = true; break; } } reader.close(); } catch (Exception e) { e.printStackTrace(); } finally { if (reader != null) { try { reader.close(); } catch (IOException ignored) { } } } //throw exception if not matched if (ok) { System.out.println("user [" + username + "] auth check ok .. "); } else { System.out.println("user [" + username + "] auth check fail .. "); throw new AuthenticationException("user [" + username + "] auth check fail .. "); } } }
编码时需要添加必要的dependency,maven配置如下,版本号需要根据自己的情况而定:
<dependency> <groupId>org.apache.hive</groupId> <artifactId>hive-jdbc</artifactId> <version>2.3.0</version> </dependency> <dependency> <groupId>org.apache.hive</groupId> <artifactId>hive-metastore</artifactId> <version>2.3.0</version> </dependency> <dependency> <groupId>org.apache.hive</groupId> <artifactId>hive-exec</artifactId> <version>2.3.0</version> </dependency> <dependency> <groupId>org.apache.hive</groupId> <artifactId>hive-service</artifactId> <version>2.3.0</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-common</artifactId> <version>2.8.1</version> </dependency>
编辑$HIVE_HOME/conf/hive-site.xml,增加或修改以下property。
其中authentication file和authentication class要根据自己的情况修改。<property> <name>hive.server2.authentication</name> <value>CUSTOM</value> <description> Expects one of [nosasl, none, ldap, kerberos, pam, custom]. Client authentication types. NONE: no authentication check LDAP: LDAP/AD based authentication KERBEROS: Kerberos/GSSAPI authentication CUSTOM: Custom authentication provider (Use with property hive.server2.custom.authentication.class) PAM: Pluggable authentication module NOSASL: Raw transport </description> </property> <property> <name>hive.server2.custom.authentication.class</name> <value>com.opticalix.CustomHiveServer2Auth</value> <description> Custom authentication class. Used when property 'hive.server2.authentication' is set to 'CUSTOM'. Provided class must be a proper implementation of the interface org.apache.hive.service.auth.PasswdAuthenticationProvider. HiveServer2 will call its Authenticate(user, passed) method to authenticate requests. The implementation may optionally implement Hadoop's org.apache.hadoop.conf.Configurable class to grab Hive's Configuration object. </description> </property> <property> <name>hive.server2.custom.authentication.file</name> <value>/usr/local/lib/hive-2.3.0/conf/hive.server2.users.conf</value> </property>
打jar包。不需要dependencies一同打进jar。这里利用maven plugin: maven-assembly-plugin(Jar打包博文见此)。添加该plugin后执行命令mvn assembly:assembly即可在target中得到jar。然后
利用scp命令将jar放在集群的$HIVE_HOME/lib,这样在hive启动时会自动加载lib中的jars。重新启动hive-server2(如果刚才的hive server进程未结束,可以用kill -9 $pid来结束进程),并使用beeline连接,此时用刚才设置的用户名/密码才能连接成功:
beeline> !connect jdbc:hive2://localhost:10000 Connecting to jdbc:hive2://localhost:10000 Enter username for jdbc:hive2://localhost:10000: opticalix Enter password for jdbc:hive2://localhost:10000: ************** ... Connected to: Apache Hive (version 2.3.0) Driver: Hive JDBC (version 1.2.1.spark2) Transaction isolation: TRANSACTION_REPEATABLE_READ
Server控制台也会输出log:
hive.server2.custom.authentication.file [/usr/local/lib/hive-2.3.0/conf/hive.server2.users.conf] .. read line=opticalix,88b6ed7f53340e0d45ff60f277e4c420 user [opticalix] auth check ok ..