Spark2.x(五十六):Queue's AM resource limit exceeded.

背景:

按照业务需求将数据拆分为60份,启动60个application分别运行对每一份数据,application的提交脚本如下:

#/bin/sh
#LANG=zh_CN.utf8
#export LANG
export SPARK_KAFKA_VERSION=0.10
export LANG=zh_CN.UTF-8
jarspath=''
for file in `ls /home/dx/pro2.0/app01/sparkjars/*.jar`
do
  jarspath=${file},$jarspath
done
jarspath=${jarspath%?}
echo $jarspath

./bin/spark-submit.sh \
--jars $jarspath \
--properties-file ../conf/spark-properties.conf \
--verbose \
--master yarn \
--deploy-mode cluster \
--name Streaming-$2-$3-$4-$5-$1-Agg-Parser \
--driver-memory 9g \
--driver-cores 1 \
--num-executors 1 \
--executor-cores 12 \
--executor-memory 22g \
--driver-java-options "-XX:+TraceClassPaths" \
--class com.dx.app01.streaming.Main \
/home/dx/pro2.0/app01/lib/app01-streaming-driver.jar $1 $2 $3 $4 $5

运行集群包含的运行节点43个节点,每个节点配置信息如下:24VCores 64G

yarn配置情况:

yarn.scheduler.minimum-allocation-mb 单个容器可申请的最小内存 1G
yarn.scheduler.maximum-allocation-mb单个容器可申请的最大内存 51G
yarn.nodemanager.resource.cpu-vcores NodeManager总的可用虚拟CPU个数 21vcores
yarn.nodemanager.resource.memory-mb 每个节点可用的最大内存,RM中的两个值不应该超过此值 51G

 

问题:

 执行上边脚本启动了60个任务,但是经过测试发现最多只能提交24个任务,然后剩余还有一个部分任务都是处于 Accepted 状态,按照目前情况至少要执行43个任务。

通过yarn node -list命令查看当前节点上运行containers情况如下:

Node-Id Node-StateNode-Http-AddressNumber-of-Running-Containers
node-53:45454RUNNINGnode-53:80421
node-62:45454RUNNINGnode-62:80424
node-44:45454RUNNINGnode-44:80423
node-37:45454RUNNINGnode-37:80420
node-35:45454RUNNINGnode-35:80421
node-07:45454RUNNINGnode-07:80420
node-30:45454RUNNINGnode-30:80420
node-56:45454RUNNINGnode-56:80422
node-47:45454RUNNINGnode-47:80420
node-42:45454RUNNINGnode-42:80422
node-03:45454RUNNINGnode-03:80426
node-51:45454RUNNINGnode-51:80422
node-33:45454RUNNINGnode-33:80421
node-04:45454RUNNINGnode-04:80421
node-48:45454RUNNINGnode-48:80426
node-39:45454RUNNINGnode-39:80420
node-60:45454RUNNINGnode-60:80421
node-54:45454RUNNINGnode-54:80420
node-45:45454RUNNINGnode-45:80420
node-63:45454RUNNINGnode-63:80421
node-09:45454RUNNINGnode-09:80421
node-01:45454RUNNINGnode-01:80421
node-36:45454RUNNINGnode-36:80423
node-06:45454RUNNINGnode-06:80420
node-61:45454RUNNINGnode-61:80421
node-31:45454RUNNINGnode-31:80420
node-40:45454RUNNINGnode-40:80420
node-57:45454RUNNINGnode-57:80421
node-59:45454RUNNINGnode-59:80421
node-43:45454RUNNINGnode-43:80421
node-52:45454RUNNINGnode-52:80421
node-34:45454RUNNINGnode-34:80421
node-38:45454RUNNINGnode-38:80420
node-50:45454RUNNINGnode-50:80424
node-46:45454RUNNINGnode-46:80421
node-08:45454RUNNINGnode-08:80421
node-55:45454RUNNINGnode-55:80421
node-32:45454RUNNINGnode-32:80420
node-41:45454RUNNINGnode-41:80422
node-05:45454RUNNINGnode-05:80421
node-02:45454RUNNINGnode-02:80421
node-58:45454RUNNINGnode-58:80420
node-49:45454RUNNINGnode-49:80420

 很明显,目前集群还有一部分节点未被使用,说明资源时充足的。

那么,至少应该能提交43个任务才对,但是目前只提交了24个任务,而且在Yarn上还提示错误信息:

[Tue Jul 30 16:33:29 +0000 2019] Application is added to the scheduler and is not yet activated. Queue's AM resource limit exceeded. Details : AM Partition = <DEFAULT_PARTITION>; 
AM Resource Request = <memory:9216MB(9G), vCores:1>; Queue Resource Limit for AM = <memory:454656MB(444G), vCores:1>; User AM Resource Limit of the queue = <memory:229376MB(224G), vCores:1>; Queue AM Resource Usage = <memory:221184MB(216G), vCores:24>;

解决方案:

其中错误日志:“Queue AM Resource Usage = <memory:221184MB(216G), vCores:24>;”中正是指目前已经运行了24个app(yarn-cluster模式下,每个app包含一个driver,driver也就是等同于AM):每个app的driver包含1个vcores,一共占用24vcores;每个app的driver内存为9G,9G*24=216G。
其中错误日志:“User AM Resource Limit of the queue = <memory:229376MB(224G), vCores:1>; ”中集群中用于运行应用程序ApplicationMaster的资源最大允许224G,这个值由参数”yarn.scheduler.capacity.maximum-am-resource-percent“决定。

yarn.scheduler.capacity.maximum-am-resource-percent

/ yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent

集群中用于运行应用程序ApplicationMaster的资源比例上限,该参数通常用于限制处于活动状态的应用程序数目。该参数类型为浮点型,默认是0.1,表示10%。

所有队列的ApplicationMaster资源比例上限可通过参数yarn.scheduler.capacity. maximum-am-resource-percent设置(可看做默认值),

而单个队列可通过参数yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent设置适合自己的值。

1)yarn.scheduler.capacity.maximum-am-resource-percent(调大)

<property>
    <!-- Maximum resources to allocate to application masters
    If this is too high application masters can crowd out actual work -->
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.5</value>
</property>

2)降低 driver 内存。

 

关于Yarn Capacity更多,更官方问题请参考官网文档:《Hadoop: Capacity Scheduler

    原文作者:spark
    原文地址: https://www.cnblogs.com/yy3b2007com/p/11273169.html
    本文转自网络文章,转载此文章仅为分享知识,如有侵权,请联系博主进行删除。
点赞