`spark.yarn.jar`和`spark.yarn.archive`的使用

2023年6月14日 318次阅读 来源: 分裂四人组

启动Spark任务时，在没有配置spark.yarn.archive或者spark.yarn.jars时， 会看到不停地上传jar，非常耗时；使用spark.yarn.archive可以大大地减少任务的启动时间，整个处理过程如下。

  1. 在本地创建zip文件
hzlishuming@hadoop691:~/env/spark$ cd jars/
hzlishuming@hadoop691:~/env/spark$ zip spark2.1.1-hadoop2.7.3.zip ./*
  1. 上传至HDFS并更改权限
hzlishuming@hadoop691:~/env/spark$ /usr/ndp/current/hdfs_client/bin/hdfs dfs -mkdir /tmp/spark-archive
hzlishuming@hadoop691:~/env/spark$ /usr/ndp/current/hdfs_client/bin/hdfs dfs -put ./spark2.1.1-hadoop2.7.3.zip /tmp/spark-archive
hzlishuming@hadoop691:~/env/spark$ /usr/ndp/current/hdfs_client/bin/hdfs dfs -chmod 775 /tmp/spark-archive/spark2.1.1-hadoop2.7.3.zip

3.配置spark-defaut.conf

spark.yarn.archive                                    hdfs:///tmp/spark-archive/spark2.1.1-hadoop2.7.3.zip

可以参考日志如下：

17/08/10 14:59:27 INFO Client: To enable the AM to login from keytab, credentials are being copied over to the AM via the YARN Secure Distributed Cache.
17/08/10 14:59:27 INFO Client: Uploading resource file:/etc/security/keytabs/hive.service.keytab -> hdfs://hz-test-01/user/hive/.sparkStaging/application_1500533600435_2825/hive.service.keytab
17/08/10 14:59:27 INFO Client: Source and destination file systems are the same. Not copying hdfs:/tmp/spark-archive/spark2.1.1-hadoop2.7.3.zip
17/08/10 14:59:27 INFO Client: Uploading resource file:/home/hzlishuming/env/spark-2.1.1/local/spark-6606333c-1e5b-462c-ad39-aaf75251c246/__spark_conf__2962372142699552959.zip -> hdfs://hz-test-01/user/hive/.sparkStaging/application_1500533600435_2825/__spark_conf__.zip
    原文作者：分裂四人组
    原文地址: https://www.jianshu.com/p/016fbd2a421b
    本文转自网络文章，转载此文章仅为分享知识，如有侵权，请联系博主进行删除。
点赞