hadoop sentry错误记录

1、报无法实例化metastore连接

hive> show tables;

FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

 

解决:

将sentry和shiro包拷贝到 hive/lib下

2、hive/conf/hive-site.xml中确实sentry服务地址配置

2019-01-16 22:10:33,005 WARN  [pool-4-thread-2]: conf.HiveAuthzConf (HiveAuthzConf.java:get(218)) – Using the deprecated config setting hive.sentry.provider instead of sentry.provider

2019-01-16 22:10:33,070 WARN  [pool-4-thread-2]: db.SimpleDBProviderBackend (SimpleDBProviderBackend.java:getPrivileges(81)) – Unable to obtain privileges from server: Property ‘sentry.service.client.server.rpc-addresses’ is missing in configuration. Will retry for 3 time(s)

2019-01-16 22:11:03,071 WARN  [pool-4-thread-2]: db.SimpleDBProviderBackend (SimpleDBProviderBackend.java:getPrivileges(81)) – Unable to obtain privileges from server: Property ‘sentry.service.client.server.rpc-addresses’ is missing in configuration. Will retry for 2 time(s)

 

解决:

需要在hive的conf/sentry-site.xml文件中添加

<property>

    <name>sentry.service.client.server.rpc-port</name>

    <value>8038</value>

</property>

<property>

    <name>sentry.service.client.server.rpc-addresses</name>

    <value> </value>

</property>

3、报sentry用户组权限异常

2019-01-16 22:18:22,474 WARN  [pool-4-thread-1]: security.UserGroupInformation (UserGroupInformation.java:doAs(1920)) – PriviledgedActionException as:biztech/hostname@BIZTECHTEST.COM (auth:KERBEROS) cause:org.apache.thrift.transport.TTransportException: Peer indicated failure: Problem with callback handler

2019-01-16 22:18:22,475 ERROR [pool-4-thread-1]: transport.SentryTransportPool (SentryTransportPool.java:getTransport(196)) – Failed to obtain transport for rsync.cm03.hadooptest.yf.ted:8038: null

2019-01-16 22:18:25,475 WARN  [pool-4-thread-1]: db.SimpleDBProviderBackend (SimpleDBProviderBackend.java:getPrivileges(81)) – Unable to obtain privileges from server: Peer indicated failure: Problem with callback handler. Will retry for 2 time(s)

 

解决:

需要在sentry/conf/sentry-site.xml文件中添加

<property>

    <name>sentry.service.admin.group</name>

    <value>biztech,hive,impala,hue,hdfs</value>

</property>

 

<property>

    <name>sentry.service.allow.connect</name>

    <value>biztech,hive,impala,hue,hdfs</value>

</property>

 

重启sentry

4、报sentry用户权限异常

hive> create table hxh(name string);

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:User biztech does not have privileges for CREATETABLE)

 

 

需要在hive的conf/sentry-site.xml文件中添加

<property>

     <name>sentry.metastore.service.users</name>

     <value>biztech,hive</value>

</property>

5、beeline kerberos登陆异常

beeline> !connect jdbc:hive2://x.x.x.x:10000/default;principal=biztech/hostname@example.COM

Error: Could not open client transport with JDBC Uri: jdbc:hive2://x.x.x.x:10000/default;principal=biztech/hostname@example.COM: Peer indicated failure: GSS initiate failed (state=08S01,code=0)

 

 

这个只能使用hiveserver本身的principal进行认证

 

解决:

vim conf/hive-env.sh中添加

export HADOOP_OPTS=”-Dsun.security.krb5.debug=true ${HADOOP_OPTS}”  #打印认证过程

 

beeline> !connect jdbc:hive2://x.x.x.x:10000/;principal=biztech/hostname@example.COM

6、hive.server2.enable.impersonation参数介绍

<property>

    <name>hive.server2.enable.impersonation</name>

    <value>true</value>

</property>

 

这个参数作用,参考链接:http://blog.51cto.com/boylook/1311584

CDH4.2.0开始支持hiveserver2的impersonation,只需要设置参数hive.server2.enable.impersonation(该参数在Hive里是HIVE_SERVER2_KERBEROS_IMPERSONATION,容易被误会和kerberos一起使用)为true即可,这样可以使得提交的HQL以提交用户来运行MAPREDUCE而不是运行Hiveserver的User,那么具体是如何实现的呢:

 

当通过ThriftCliService开启一个Session时,如果开启了上面的参数,则会调用CliService的openSessionWithImpersonation方法进而SessionManager就会生成一个HiveSession的动态代理HiveSessionImplwithUGI,并把用户的UGI传入进去:

 

session = (HiveSession)HiveSessionProxy.getProxy(hiveSessionUgi,hiveSessionUgi.getSessionUgi());

 

hiveSessionUgi.setProxySession(session);

 

每次执行方法的时候都是通过ShimLoader.getHadoopShims().doAs执行,如果没有指定该参数则是以运行Hiveserver的用户执行.

7、报sentry访问找不到

2019-01-18 14:58:18,946 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.sentry.hdfs.SentryAuthorizationProvider not found

        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2311)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:1250)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:706)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:692)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:844)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:823)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1547)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1615)

Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.sentry.hdfs.SentryAuthorizationProvider not found

        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2279)

        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2303)

        … 7 more

Caused by: java.lang.ClassNotFoundException: Class org.apache.sentry.hdfs.SentryAuthorizationProvider not found

        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2185)

        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2277)

        … 8 more

 

解决:

拷贝 sentry服务lib下的sentry*和shiro-core-1.2.3.jar到hadoop/lib/sentry目录下

 

vim etc/hadoop/hadoop-env.sh

export HADOOP_CLASSPATH=${HADOOP_HOME}/lib/*:${HADOOP_HOME}/lib/sentry/*:$HADOOP_CLASSPATH

8、org.apache.commons.pool2.KeyedPooledObjectFactory类找不到

2019-01-18 15:03:08,154 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.

java.lang.NoClassDefFoundError: org/apache/commons/pool2/KeyedPooledObjectFactory

        at org.apache.sentry.hdfs.SentryHDFSServiceClientFactory.<init>(SentryHDFSServiceClientFactory.java:71)

        at org.apache.sentry.hdfs.SentryHDFSServiceClientFactory.create(SentryHDFSServiceClientFactory.java:60)

        at org.apache.sentry.hdfs.SentryUpdater.getUpdates(SentryUpdater.java:41)

        at org.apache.sentry.hdfs.SentryAuthorizationInfo.update(SentryAuthorizationInfo.java:132)

        at org.apache.sentry.hdfs.SentryAuthorizationInfo.start(SentryAuthorizationInfo.java:251)

        at org.apache.sentry.hdfs.SentryAuthorizationProvider.start(SentryAuthorizationProvider.java:130)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:1254)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:706)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:692)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:844)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:823)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1547)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1615)

Caused by: java.lang.ClassNotFoundException: org.apache.commons.pool2.KeyedPooledObjectFactory

        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

        at java.security.AccessController.doPrivileged(Native Method)

        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

        … 13 more

 

下载:commons-pool2-2.0.jar到hadoop/lib下

9、报org.apache.thrift.transport.TTransport类找不到

2019-01-18 15:08:02,328 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.

java.lang.NoClassDefFoundError: org/apache/thrift/transport/TTransport

        at org.apache.sentry.hdfs.SentryHDFSServiceClientFactory.<init>(SentryHDFSServiceClientFactory.java:71)

        at org.apache.sentry.hdfs.SentryHDFSServiceClientFactory.create(SentryHDFSServiceClientFactory.java:60)

        at org.apache.sentry.hdfs.SentryUpdater.getUpdates(SentryUpdater.java:41)

        at org.apache.sentry.hdfs.SentryAuthorizationInfo.update(SentryAuthorizationInfo.java:132)

        at org.apache.sentry.hdfs.SentryAuthorizationInfo.start(SentryAuthorizationInfo.java:251)

        at org.apache.sentry.hdfs.SentryAuthorizationProvider.start(SentryAuthorizationProvider.java:130)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:1254)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:706)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:692)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:844)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:823)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1547)

        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1615)

Caused by: java.lang.ClassNotFoundException: org.apache.thrift.transport.TTransport

        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)

        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)

        at java.security.AccessController.doPrivileged(Native Method)

        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

        … 13 more

 

下载:libthrift-0.11.0.jar到hadoop/lib下

10、报org.apache.sentry.hdfs.MetastorePlugin类找不到

MetaException(message:Failed to instantiate listener named: org.apache.sentry.binding.metastore.SentryMetastorePostEventListener, reason: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.sentry.hdfs.MetastorePlugin not found)

           at org.apache.hadoop.hive.metastore.MetaStoreUtils.getMetaStoreListeners(MetaStoreUtils.java:1514)

           at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:555)

           at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)

           at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)

           at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6475)

           at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6470)

           at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6720)

           at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6647)

           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

           at java.lang.reflect.Method.invoke(Method.java:606)

           at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

           at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Exception in thread “main” MetaException(message:Failed to instantiate listener named: org.apache.sentry.binding.metastore.SentryMetastorePostEventListener, reason: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.sentry.hdfs.MetastorePlugin not found)

           at org.apache.hadoop.hive.metastore.MetaStoreUtils.getMetaStoreListeners(MetaStoreUtils.java:1514)

           at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:555)

           at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)

           at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)

           at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6475)

           at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6470)

           at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6720)

           at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6647)

           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

           at java.lang.reflect.Method.invoke(Method.java:606)

           at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

           at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

 

 

解决:

https://jar-download.com/download-handling.php

 

搜索:org.apache.sentry.hdfs.MetastorePlugin

 

替换:sentry-hdfs-service-1.7.1.jar 包

11、beeline创建role报sentry principal异常

0: jdbc:hive2://x.x.x.x:10000/> create role admin_role;

INFO  : Compiling command(queryId=biztech_20190121183939_1e3d3db7-64a4-4cc1-b404-867fdaa12c0f): create role admin_role

INFO  : Semantic Analysis Completed

INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)

INFO  : Completed compiling command(queryId=biztech_20190121183939_1e3d3db7-64a4-4cc1-b404-867fdaa12c0f); Time taken: 0.577 seconds

INFO  : Executing command(queryId=biztech_20190121183939_1e3d3db7-64a4-4cc1-b404-867fdaa12c0f): create role admin_role

INFO  : Starting task [Stage-0:DDL] in serial mode

ERROR : Error processing Sentry command: Property ‘sentry.service.server.principal’ is missing in configuration

ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Property ‘sentry.service.server.principal’ is missing in configuration

INFO  : Completed executing command(queryId=biztech_20190121183939_1e3d3db7-64a4-4cc1-b404-867fdaa12c0f); Time taken: 0.05 seconds

Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Property ‘sentry.service.server.principal’ is missing in configuration (state=08S01,code=1)

 

 

这个需要在hive/conf/sentry-site.xml文件中添加

<property>

    <name>hive.sentry.server</name>

    <value>servername</value>

</property>

<property>

    <name>sentry.service.security.mode</name>

    <value>kerberos</value>

</property>

 

<property>

    <name>sentry.service.server.principal</name>

    <value> </value>

</property>

 

<property>

    <name>sentry.service.server.keytab</name>

    <value> </value>

</property>

12、beeline创建role的时候报sequence_table主键长度异常

0: jdbc:hive2://x.x.x.x:10000/> create role admin_role;

INFO  : Compiling command(queryId=biztech_20190121184444_84c7e88e-00fb-41e6-beaf-ef144a9b855a): create role admin_role

INFO  : Semantic Analysis Completed

INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)

INFO  : Completed compiling command(queryId=biztech_20190121184444_84c7e88e-00fb-41e6-beaf-ef144a9b855a); Time taken: 0.105 seconds

INFO  : Executing command(queryId=biztech_20190121184444_84c7e88e-00fb-41e6-beaf-ef144a9b855a): create role admin_role

INFO  : Starting task [Stage-0:DDL] in serial mode

ERROR : Error processing Sentry command: null

ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. null

INFO  : Completed executing command(queryId=biztech_20190121184444_84c7e88e-00fb-41e6-beaf-ef144a9b855a); Time taken: 369.295 seconds

Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. null (state=08S01,code=1)

 

2019-01-22 14:48:01,507 ERROR [HiveServer2-Background-Pool: Thread-78]: operation.Operation (SQLOperation.java:run(304)) – Error running hive query:

org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. null

           at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400)

           at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238)

           at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)

           at org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)

           at java.security.AccessController.doPrivileged(Native Method)

           at javax.security.auth.Subject.doAs(Subject.java:415)

           at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)

           at org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)

           at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

           at java.util.concurrent.FutureTask.run(FutureTask.java:262)

           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

           at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.reflect.UndeclaredThrowableException

           at com.sun.proxy.$Proxy25.createRole(Unknown Source)

           at org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask.processRoleDDL(SentryGrantRevokeTask.java:223)

           at org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask.execute(SentryGrantRevokeTask.java:127)

           at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)

           at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:99)

           at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2052)

           at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1748)

           at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)

           at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)

           at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)

           at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)

           … 11 more

Caused by: java.lang.reflect.InvocationTargetException

           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

           at java.lang.reflect.Method.invoke(Method.java:606)

           at org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:95)

           at org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41)

           … 22 more

Caused by: java.lang.RuntimeException: Unknown error for request: TCreateSentryRoleRequest(protocol_version:2, requestorUserName:biztech, roleName:admin_role), message: The transaction has reached max retry number, Exception thrown calling table.exists() for `SEQUENCE_TABLE`. Server Stacktrace: java.lang.Exception: The transaction has reached max retry number, Exception thrown calling table.exists() for `SEQUENCE_TABLE`

           at org.apache.sentry.provider.db.service.persistent.TransactionManager$ExponentialBackoff.execute(TransactionManager.java:257)

           at org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionWithRetry(TransactionManager.java:188)

           at org.apache.sentry.provider.db.service.persistent.SentryStore.createSentryRole(SentryStore.java:353)

           at org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.create_sentry_role(SentryPolicyStoreProcessor.java:201)

           at org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$create_sentry_role.getResult(SentryPolicyService.java:957)

           at org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$create_sentry_role.getResult(SentryPolicyService.java:942)

           at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38)

           at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)

           at org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:36)

           at org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:134)

           at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:291)

           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

           at java.lang.Thread.run(Thread.java:745)

Caused by: javax.jdo.JDOException: Exception thrown calling table.exists() for `SEQUENCE_TABLE`

NestedThrowables:

com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 767 bytes

           at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)

           at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)

           at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)

           at org.apache.sentry.provider.db.service.persistent.SentryStore$1.execute(SentryStore.java:361)

           at org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:123)

           at org.apache.sentry.provider.db.service.persistent.TransactionManager$1.call(TransactionManager.java:192)

           at org.apache.sentry.provider.db.service.persistent.TransactionManager$ExponentialBackoff.execute(TransactionManager.java:233)

           … 13 more

Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Specified key was too long; max key length is 767 bytes

           at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown Source)

           at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

           at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

           at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)

           at com.mysql.jdbc.Util.getInstance(Util.java:386)

           at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1052)

           at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)

           at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)

           at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)

           at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)

           at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)

           at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2569)

           at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:824)

           at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:667)

           at com.jolbox.bonecp.StatementHandle.execute(StatementHandle.java:300)

           at org.datanucleus.store.rdbms.table.AbstractTable.executeDdlStatement(AbstractTable.java:760)

           at org.datanucleus.store.rdbms.table.AbstractTable.executeDdlStatementList(AbstractTable.java:711)

           at org.datanucleus.store.rdbms.table.AbstractTable.create(AbstractTable.java:425)

           at org.datanucleus.store.rdbms.table.AbstractTable.exists(AbstractTable.java:488)

           at org.datanucleus.store.rdbms.valuegenerator.TableGenerator.repositoryExists(TableGenerator.java:242)

           at org.datanucleus.store.rdbms.valuegenerator.AbstractRDBMSGenerator.obtainGenerationBlock(AbstractRDBMSGenerator.java:86)

           at org.datanucleus.store.valuegenerator.AbstractGenerator.obtainGenerationBlock(AbstractGenerator.java:197)

           at org.datanucleus.store.valuegenerator.AbstractGenerator.next(AbstractGenerator.java:105)

           at org.datanucleus.store.rdbms.RDBMSStoreManager.getStrategyValueForGenerator(RDBMSStoreManager.java:2033)

           at org.datanucleus.store.AbstractStoreManager.getStrategyValue(AbstractStoreManager.java:1386)

           at org.datanucleus.ExecutionContextImpl.newObjectId(ExecutionContextImpl.java:3827)

           at org.datanucleus.state.JDOStateManager.setIdentity(JDOStateManager.java:2571)

           at org.datanucleus.state.JDOStateManager.initialiseForPersistentNew(JDOStateManager.java:513)

           at org.datanucleus.state.ObjectProviderFactoryImpl.newForPersistentNew(ObjectProviderFactoryImpl.java:232)

           at org.datanucleus.ExecutionContextImpl.newObjectProviderForPersistentNew(ExecutionContextImpl.java:1414)

           at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2218)

           at org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:2065)

           at org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1913)

           at org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217)

           at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:727)

           … 18 more

 

分析:

参考链接:https://blog.csdn.net/bigdataf/article/details/78921409

mysql建立索引时,数据库计算key的长度是累加所有Index用到的字段的char长度后再按下面比例乘起来不能超过限定的key长度1000:

latin1 = 1 byte = 1 character

uft8 = 3 byte = 1 character

gbk = 2 byte = 1 character

举例能看得更明白些,以GBK为例:

CREATE UNIQUE INDEX unique_record ON reports (report_name, report_client, report_city);

其中report_name varchar(200), report_client varchar(200), report_city varchar(200)

(200 + 200 +200) * 2 = 1200 > 1000,所有就会报1071错误,只要将report_city改为varchar(100)那么索引就能成功建立

 

解决:

修改数据库配置:

[mysqld]

character-set-server = latin1

character_set_server = latin1

default-storage-engine=INNODB

innodb_large_prefix=on

重启数据库

 

注意:

这里数据库是5.5的

 

重新创建hive、sentry数据库

重新初始化hive、sentry数据库

再次执行create role成功

 

注意:

这里初始化hive数据库后,一定需要删除sentry数据库,然后初始化一下,否则还会有上面的错误。

 

点赞