HBase/Spark Guava依赖冲突解决方案

2023年1月16日 582次阅读 来源: zfylin

前言

版本guava 版本
Spark2.2.012.0.1
HBase1.0.0-cdh5.6.020.0

由于guava的版本在16.0以后，不向后兼容，所以Spark程序中集成HBase是会报找不到依赖的方法错误， 如下：

Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.IllegalAccessError: tried to access method com.google.common.base.Stopwatch.<init>()V from class org.apache.hadoop.hbase.zookeeper.MetaTableLocator
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:229)
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:202)
    at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:326)
    at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:301)
    at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:166)
    at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:161)
    at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794)
    at org.springframework.data.hadoop.hbase.HbaseTemplate$1.doInTable(HbaseTemplate.java:132)
    at org.springframework.data.hadoop.hbase.HbaseTemplate.execute(HbaseTemplate.java:61)
    ... 75 more

解决方案

使用 maven-shade-plugin 打包hbase依赖，把依赖的包打包进项目，并且修改依赖包的包名，这样hbase引用的就是修改包名后的guava包了，不在依赖原生的guava包。

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <parent>
        <artifactId>xql</artifactId>
        <groupId>com.xxx.xxx.xxx</groupId>
        <version>1.0.0</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>

    <artifactId>your-hbase</artifactId>

    <dependencies>
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-server</artifactId>
            <version>1.0.0</version>
            <exclusions>
                <exclusion>
                    <groupId>com.google.guava</groupId>
                    <artifactId>guava</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-common</artifactId>
            <version>1.0.0</version>
        </dependency>
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>16.0</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.0.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                    <!-- Additional configuration. -->
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

然后，spark集成hbase时，引用maven-shade-plugin打包后的hbase包，冲突解决。

 <dependency>
     <groupId>com.xxx.xxx.xxx</groupId>
     <artifactId>your-hbase</artifactId>
     <version>1.0.0</version>
 </dependency>
    原文作者：zfylin
    原文地址: https://www.jianshu.com/p/8422ba13f096
    本文转自网络文章，转载此文章仅为分享知识，如有侵权，请联系博主进行删除。
点赞