异常问题：Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.
spark-submit提交脚本：
[spark@master work]$ more submit.sh #! /bin/bash jars="" for file in `ls /home/spark/work/jars/*.jar` do jars=$file,$jars #echo $jars done echo "------------------------------------" echo $jars echo "------------------------------------" /opt/spark-2.2.1-bin-hadoop2.7/bin/spark-submit \ --jars $jars \ --master yarn \ --verbose \ --driver-java-options "-XX:+TraceClassPaths" \ --num-executors 2 \ --executor-memory 1G \ --executor-cores 1 \ --driver-memory 1G \ --class com.dx.streaming.producer.TestProducer \ /home/spark/work/kafka-streaming-test.jar
在master节点上提交后，抛出异常：
18/07/09 05:00:16 INFO yarn.Client: Application report for application_1531069155353_0001 (state: ACCEPTED) 18/07/09 05:00:17 INFO yarn.Client: Application report for application_1531069155353_0001 (state: FAILED) 18/07/09 05:00:17 INFO yarn.Client: client token: N/A diagnostics: Application application_1531069155353_0001 failed 2 times due to AM Container for appattempt_1531069155353_0001_000002 exited with exitCode: -103 Failing this attempt.Diagnostics: [2018-07-09 05:00:16.132]Container [pid=5674,containerID=container_e22_1531069155353_0001_02_000001] is running beyond virtual memory limits. Current usage: 195.8 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container. Dump of the process-tree for container_e22_1531069155353_0001_02_000001 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 5674 5673 5674 5674 (bash) 0 0 115896320 350 /bin/bash -c /opt/jdk1.8.0_171/bin/java -server -Xmx512m -Djava.io.tmpdir=/opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/tmp -Dspark.yarn.app.container.log.dir=/opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg '192.168.0.120:40486' --properties-file /opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/__spark_conf__/__spark_conf__.properties 1> /opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/stdout 2> /opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/stderr |- 5687 5674 5674 5674 (java) 208 35 2323283968 49770 /opt/jdk1.8.0_171/bin/java -server -Xmx512m -Djava.io.tmpdir=/opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/tmp -Dspark.yarn.app.container.log.dir=/opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg 192.168.0.120:40486 --properties-file /opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/__spark_conf__/__spark_conf__.properties [2018-07-09 05:00:16.133] [2018-07-09 05:00:16.152]Container killed on request. Exit code is 143[2018-07-09 05:00:16.152] [2018-07-09 05:00:16.156]Container exited with a non-zero exit code 143. [2018-07-09 05:00:16.156] For more detailed output, check the application tracking page: http://master:8088/cluster/app/application_1531069155353_0001 Then click on links to logs of each attempt. . Failing the application. ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1531083596749 final status: FAILED tracking URL: http://master:8088/cluster/app/application_1531069155353_0001 user: spark 18/07/09 05:00:17 INFO yarn.Client: Deleted staging directory hdfs://HA/user/spark/.sparkStaging/application_1531069155353_0001 18/07/09 05:00:17 ERROR spark.SparkContext: Error initializing SparkContext. org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173) at org.apache.spark.SparkContext.<init>(SparkContext.scala:509) ▽ at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910) at com.dx.streaming.producer.TestProducer.main(TestProducer.java:39) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 18/07/09 05:00:17 INFO server.AbstractConnector: Stopped Spark@60d84f61{HTTP/1.1,[http/1.1]}{0.0.0.0:4040} 18/07/09 05:00:17 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.0.120:4040 18/07/09 05:00:17 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered! 18/07/09 05:00:17 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors 18/07/09 05:00:17 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down 18/07/09 05:00:17 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices (serviceOption=None, services=List(), started=false) 18/07/09 05:00:17 INFO cluster.YarnClientSchedulerBackend: Stopped 18/07/09 05:00:17 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 18/07/09 05:00:17 INFO memory.MemoryStore: MemoryStore cleared 18/07/09 05:00:17 INFO storage.BlockManager: BlockManager stopped 18/07/09 05:00:17 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 18/07/09 05:00:17 WARN metrics.MetricsSystem: Stopping a MetricsSystem that is not running 18/07/09 05:00:17 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 18/07/09 05:00:17 INFO spark.SparkContext: Successfully stopped SparkContext Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173) at org.apache.spark.SparkContext.<init>(SparkContext.scala:509) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910) at com.dx.streaming.producer.TestProducer.main(TestProducer.java:39) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 18/07/09 05:00:17 INFO util.ShutdownHookManager: Shutdown hook called 18/07/09 05:00:17 INFO util.ShutdownHookManager: Deleting directory /opt/spark-2.2.1-bin-hadoop2.7/spark-255c65cf-19d8-481a-94bc-843e058eb669
错误原因：
Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.
YARN的虚拟内存计算方式导致，上例中用户程序申请的内存为1Gb，YARN根据此值乘以一个比例（默认为2.1）得出申请的虚拟内存的值，当YARN计算的用户程序所需虚拟内存值大于计算出来的值时，就会报出以上错误。
解决方案：
上网一查，发现有两种方法能解决这个问题:
1.将yarn.nodemanager.vmem-check-enabled的值改为false，即不检查VM的值；（修改yarn-site.xml）参考《https://discuss.elastic.co/t/unable-to-start-elasticsearch-yarn-container-is-running-beyond-virtual-memory-limits/26102/6》
<property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> </property>
2.将yarn.scheduler.minimum-allocation-mb的值调高一些，默认是1024mb,或者修改yarn.nodemanager.vmem-pmem-ratio的值，默认为2.1，将该值改得更大。
于是，这里我采用了第一种方法，关闭vn检查，需要重新格式化namenode，参考《Kafka：ZK+Kafka+Spark Streaming集群环境搭建（十）安装hadoop2.9.0搭建HA》，再次提交，果然好了。终于解决了这个困扰一天的问题啦！
