一次Spark上包依赖冲突的解决

2023年12月14日 320次阅读 来源: 虎耳

本地环境IDEA, 但把Jar包上传到Spark集群的时候, Job会失败. 错误信息

2017-03-29 18:12:48,190 ERROR [Driver] yarn.ApplicationMaster (Logging.scala:logError(95)) - User class threw exception: java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
    at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.java:192)
    at org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:133)
    at com.dtise.common.elasticsearch.ESClientSingleton.getClientWithHost(ESClientSingleton.java:39)
    at com.dtise.common.elasticsearch.ESClientSingleton.getTestClient(ESClientSingleton.java:29)
    at com.dtise.common.elasticsearch.ESClientProvider.getTestClient(ESClientProvider.java:22)
    at com.dtise.data.batch.bigtable.ElasticHelper$.createIndexWithMapping(ElasticHelper.scala:70)
    at com.dtise.data.batch.bigtable.ElasticHelper$.importToEs(ElasticHelper.scala:19)
    at com.dtise.data.batch.bigtable.BigTablePipeline$.start(BigTablePipeline.scala:22)
    at com.dtise.data.batch.bigtable.BigTableGo$.main(BigTableGo.scala:57)
    at com.dtise.data.batch.bigtable.BigTableGo.main(BigTableGo.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
  • 发现是Guava版本的问题, 运行时Guava的版本低.
  • 我们要的是18.0的版本, 于是将pom依赖中所有低于18.0的都exclude掉, 但还是不行.
    关于查看依赖, 用的以下两种方法:
    • 可以用mvn dependency:tree -Dverbose | grep guava查看, 这个方便一下子搜索出来. 记得加上-Dverbose
    • 用IDEA的话, 通过Maven Project tab里面的Show Dependencies,可以看到图形化的包依赖.可以直接exclude不合的包.
  • 看cdh上oozie job的log, 有这么一句guava-14.0.1.jar -> resource { scheme: "hdfs" host: "nameservice1" port: -1 file: "/user/oozie/share/lib/lib_20170216215144/spark/guava-14.0.1.jar" } size: 2189117 timestamp: 1487253111175 type: FILE visibility: PUBLIC
    • 在/user/oozie/share/lib/lib_20170216215144/spark里面找到一个guava-14.0.1.jar, 将其改名, 并上传一个guava-18.0.jar
    • 但还是不行, oozie里面记住了之前的版本, 还是报错找不到guava-14.0.1.jar. 为什么oozie会记住用某个版本的jar还有待调查.
    • 将guava-18.0.jar改名成guava-14.0.1.jar. 问题解决.
    原文作者：虎耳
    原文地址: https://www.jianshu.com/p/08fbdd2af48f
    本文转自网络文章，转载此文章仅为分享知识，如有侵权，请联系博主进行删除。
点赞