Spark SQL User Impersonation 功能需要在hive-site.xml 中把hive.server2.enable.doAs设置为true
我们使用的版本是
spark 2.4
hadoop 2.7.2
hive 1.2.1
我们来重现这个问题
- 我们先来启动sts
./sbin/start-thriftserver.sh --driver-memory 1G \
--executor-memory 1G --num-executors 1 \
--master yarn --deploy-mode client \
--conf "spark.executor.extraJavaOptions=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,a
ddress=4002" \
--driver-java-options "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=4001"
我们使用了yarn的模式启动的sts ,并且driver和executor都开了个远程调式的端口
- 使用beeline 连接上sts , 并且创建一张表 . 插入一条数据进去
./beeline -u jdbc:hive2://localhost:10000 -n james.xu
0: jdbc:hive2://localhost:10000> create table tmp.ttt ( name string , age int);
0: jdbc:hive2://localhost:10000> insert into tmp.ttt values ('rrr', 66) ;
在insert到tmp.ttt 这张测试表时就报错了
org.apache.hadoop.security.AccessControlException: Permission denied: user=james.xu, access=WRITE, inode="/user/hadoop/warehouse/tmp.db/ttt/.hive-staging_hive_2019-02-28_15-26-26_543_1953699866921090093-2/-ext-10000/_temporary/0/task_20190228152645_0001_m_000000/part-00000-c688ee6c-6d01-47d9-a4f4-c3bad434e3fe-c000":hadoop:supergroup:drwxrwxr-x
出错时的截图
问题分析
- spark sql 在向目标表写数据的时候 , 默认会在对应的目录下会产生.hive-staging的临时文件. 当临时文件写完后 . driver在做commit job时. 会将.hive-staging这个临时目录下的数据文件move到tmp.db/ttt 下面. 上面截图中堆栈报错的信息 就是在最后做commit job 时报错了.
我们看下面下ttt这个目录下所有文件的权限, 前面几个临时文件权限貌似没问题 . 后面几个临时文件的权限的owner居然是hadoop这个超级用户的.
hdfs中文件权限
貌似问题出现在这里.so , 去先去driver中创建临时文件夹的地方打个断点看下
熟悉hadoop的同学知道 , 我们在org.apache.hadoop.hdfs.DFSClient primitiveMkdir 创建文件夹这个方法中打个断点 远程连上sts 的driver
staging第一层目录创建堆栈
从上面图片的方法调用堆栈看到了doAs , 创建staging第一层目录时有userGroupInfomation的信息所以没问题
那我们继续在executor中创建文件的地方打个断点往下看
image.png
详细的具体堆栈如下
Breakpoint reached at org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:80)
Breakpoint reached
at org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:80)
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:261)
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:246)
at org.apache.spark.sql.hive.execution.HiveOutputWriter.<init>(HiveFileFormat.scala:123)
at org.apache.spark.sql.hive.execution.HiveFileFormat\$\$anon\$1.newInstance(HiveFileFormat.scala:103)
at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:120)
at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:108)
at org.apache.spark.sql.execution.datasources.FileFormatWriter\$.org\$apache\$spark\$sql\$execution\$datasources\$FileFormatWriter\$\$executeTask(FileFormatWriter.scala:233)
at org.apache.spark.sql.execution.datasources.FileFormatWriter\$\$anonfun\$write\$1.apply(FileFormatWriter.scala:169)
at org.apache.spark.sql.execution.datasources.FileFormatWriter\$\$anonfun\$write\$1.apply(FileFormatWriter.scala:168)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor\$TaskRunner\$\$anonfun\$10.apply(Executor.scala:402)
at org.apache.spark.util.Utils\$.tryWithSafeFinally(Utils.scala:1360)
at org.apache.spark.executor.Executor\$TaskRunner.run(Executor.scala:408)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor\$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
executor 端创建文件(_temporary/0/_temporary) 的是在方法HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter中完成的 , 在调用方法的堆栈中的中没有看到doAs了 , 也就没有了UserGroupInformation 的信息 .
此时临时的数据文件是以hadoop超级用户写进去的. 然后在driver端做commit job 时. 又是用James.xu 这个用户.此时就会出现一开始的那个报错信息. Premission denied.
hive mr 引擎分析
那为什么用 hive的 mr的引擎 就没有这个问题呢? 那继续在hive 的mr中打个断点看下
在hive 的thrift server 中想要调式reduce 的话需要加以下的参数 , 注意suspend=y, 需要阻塞reduce端程序的运行等远程debug上去
set mapreduce.reduce.java.opts -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=4002
insert into tmp.ttt select "222" , count(1) from dev.af_student
同样在创建文件的地方打了个断点下面是详细的调用堆栈信息.
Breakpoint reached at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:591)
Breakpoint reached
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:591)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:566)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:675)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1016)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1043)
at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1092)
at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:616)
at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:278)
at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:453)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(AccessController.java:-1)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
image.png
我们看到在YarnChild运行时中就会带上UserGroupInformation的信息. 所以hive 的mr引擎没有这个问题.
总结
hive 的mr 引擎在启用 user impersonation 功能时没问题的原因是 在reduce端运行的程序 由 yarn child 起调时就已经加上了userGroupInformation的信息. 所以对于文件的操作权限没问题. 而spark 的thrift server 中executor在写临时文件的数据文件时 没有doAs 也就没有UGI的信息. 而是以本身起调程序的用户来读写hdfs上的文件的, 而diver端读写hdfs文件又是由当前这个submit user 来完成的. 这两端的用户不匹配 而导致的问题.所以就出现文章开头的permission denied .在spark社区也看到了类似问题的issues
