hadoop sentry错误记录

2023年1月22日 244次阅读

1、报无法实例化metastore连接

hive> show tables;

FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

解决：

将sentry和shiro包拷贝到 hive/lib下

2、hive/conf/hive-site.xml中确实sentry服务地址配置

2019-01-16 22:10:33,005 WARN [pool-4-thread-2]: conf.HiveAuthzConf (HiveAuthzConf.java:get(218)) – Using the deprecated config setting hive.sentry.provider instead of sentry.provider

2019-01-16 22:10:33,070 WARN [pool-4-thread-2]: db.SimpleDBProviderBackend (SimpleDBProviderBackend.java:getPrivileges(81)) – Unable to obtain privileges from server: Property ‘sentry.service.client.server.rpc-addresses’ is missing in configuration. Will retry for 3 time(s)

2019-01-16 22:11:03,071 WARN [pool-4-thread-2]: db.SimpleDBProviderBackend (SimpleDBProviderBackend.java:getPrivileges(81)) – Unable to obtain privileges from server: Property ‘sentry.service.client.server.rpc-addresses’ is missing in configuration. Will retry for 2 time(s)

解决：

需要在hive的conf/sentry-site.xml文件中添加

<name>sentry.service.client.server.rpc-port</name>

</property>

<name>sentry.service.client.server.rpc-addresses</name>

</property>

3、报sentry用户组权限异常

2019-01-16 22:18:22,474 WARN [pool-4-thread-1]: security.UserGroupInformation (UserGroupInformation.java:doAs(1920)) – PriviledgedActionException as:biztech/hostname@BIZTECHTEST.COM (auth:KERBEROS) cause:org.apache.thrift.transport.TTransportException: Peer indicated failure: Problem with callback handler

2019-01-16 22:18:22,475 ERROR [pool-4-thread-1]: transport.SentryTransportPool (SentryTransportPool.java:getTransport(196)) – Failed to obtain transport for rsync.cm03.hadooptest.yf.ted:8038: null

2019-01-16 22:18:25,475 WARN [pool-4-thread-1]: db.SimpleDBProviderBackend (SimpleDBProviderBackend.java:getPrivileges(81)) – Unable to obtain privileges from server: Peer indicated failure: Problem with callback handler. Will retry for 2 time(s)

解决：

需要在sentry/conf/sentry-site.xml文件中添加

<name>sentry.service.admin.group</name>

<value>biztech,hive,impala,hue,hdfs</value>

</property>

<name>sentry.service.allow.connect</name>

<value>biztech,hive,impala,hue,hdfs</value>

</property>

重启sentry

4、报sentry用户权限异常

hive> create table hxh(name string);

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:User biztech does not have privileges for CREATETABLE)

需要在hive的conf/sentry-site.xml文件中添加

<name>sentry.metastore.service.users</name>

<value>biztech,hive</value>

</property>

5、beeline kerberos登陆异常

beeline> !connect jdbc:hive2://x.x.x.x:10000/default;principal=biztech/hostname@example.COM

Error: Could not open client transport with JDBC Uri: jdbc:hive2://x.x.x.x:10000/default;principal=biztech/hostname@example.COM: Peer indicated failure: GSS initiate failed (state=08S01,code=0)

这个只能使用hiveserver本身的principal进行认证

解决：

vim conf/hive-env.sh中添加

export HADOOP_OPTS=”-Dsun.security.krb5.debug=true ${HADOOP_OPTS}” #打印认证过程

beeline> !connect jdbc:hive2://x.x.x.x:10000/;principal=biztech/hostname@example.COM

6、hive.server2.enable.impersonation参数介绍

<name>hive.server2.enable.impersonation</name>

</property>

这个参数作用，参考链接：http://blog.51cto.com/boylook/1311584

CDH4.2.0开始支持hiveserver2的impersonation,只需要设置参数hive.server2.enable.impersonation(该参数在Hive里是HIVE_SERVER2_KERBEROS_IMPERSONATION，容易被误会和kerberos一起使用)为true即可，这样可以使得提交的HQL以提交用户来运行MAPREDUCE而不是运行Hiveserver的User，那么具体是如何实现的呢：

当通过ThriftCliService开启一个Session时，如果开启了上面的参数，则会调用CliService的openSessionWithImpersonation方法进而SessionManager就会生成一个HiveSession的动态代理HiveSessionImplwithUGI，并把用户的UGI传入进去：

session = (HiveSession)HiveSessionProxy.getProxy(hiveSessionUgi,hiveSessionUgi.getSessionUgi());

hiveSessionUgi.setProxySession(session);

每次执行方法的时候都是通过ShimLoader.getHadoopShims().doAs执行，如果没有指定该参数则是以运行Hiveserver的用户执行.

7、报sentry访问找不到

2019-01-18 14:58:18,946 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.sentry.hdfs.SentryAuthorizationProvider not found

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2311)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:1250)

at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:706)

at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:692)

at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:844)

at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:823)

at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1547)

at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1615)

Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.sentry.hdfs.SentryAuthorizationProvider not found

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2279)

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2303)

… 7 more

Caused by: java.lang.ClassNotFoundException: Class org.apache.sentry.hdfs.SentryAuthorizationProvider not found

at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2185)

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2277)

… 8 more

解决：

拷贝 sentry服务lib下的sentry*和shiro-core-1.2.3.jar到hadoop/lib/sentry目录下

vim etc/hadoop/hadoop-env.sh

export HADOOP_CLASSPATH=${HADOOP_HOME}/lib/*:${HADOOP_HOME}/lib/sentry/*:$HADOOP_CLASSPATH

8、org.apache.commons.pool2.KeyedPooledObjectFactory类找不到

2019-01-18 15:03:08,154 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.

java.lang.NoClassDefFoundError: org/apache/commons/pool2/KeyedPooledObjectFactory

at org.apache.sentry.hdfs.SentryHDFSServiceClientFactory.<init>(SentryHDFSServiceClientFactory.java:71)

at org.apache.sentry.hdfs.SentryHDFSServiceClientFactory.create(SentryHDFSServiceClientFactory.java:60)

at org.apache.sentry.hdfs.SentryUpdater.getUpdates(SentryUpdater.java:41)

at org.apache.sentry.hdfs.SentryAuthorizationInfo.update(SentryAuthorizationInfo.java:132)

at org.apache.sentry.hdfs.SentryAuthorizationInfo.start(SentryAuthorizationInfo.java:251)

at org.apache.sentry.hdfs.SentryAuthorizationProvider.start(SentryAuthorizationProvider.java:130)

at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:1254)

at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:706)

at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:692)

at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:844)

at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:823)

at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1547)

at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1615)

Caused by: java.lang.ClassNotFoundException: org.apache.commons.pool2.KeyedPooledObjectFactory

at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

… 13 more

下载：commons-pool2-2.0.jar到hadoop/lib下

9、报org.apache.thrift.transport.TTransport类找不到

2019-01-18 15:08:02,328 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.

java.lang.NoClassDefFoundError: org/apache/thrift/transport/TTransport