apache-spark – 通过spark-submit将JAR提交给Spark时的ClassNotFoundException

2023年11月16日 187次阅读

我正在努力使用spark-submit向Apache Spark提交JAR.

为了方便起见,我尝试使用这个blog post.代码是

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf

object SimpleScalaSpark { 
  def main(args: Array[String]) {
    val logFile = "/Users/toddmcgrath/Development/spark-1.6.1-bin-hadoop2.4/README.md" // I've replaced this with the path to an existing file
    val conf = new SparkConf().setAppName("Simple Application").setMaster("local[*]")
    val sc = new SparkContext(conf)
    val logData = sc.textFile(logFile, 2).cache()
    val numAs = logData.filter(line => line.contains("a")).count()
    val numBs = logData.filter(line => line.contains("b")).count()
    println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
  }
}

我正在运行Intellij Idea 2017.1并在Spark 2.1.0上运行.当我在IDE中运行它时,一切运行正常.

然后我将其构建为JAR并尝试使用spark-submit,如下所示

./spark-submit --class SimpleScalaSpark --master local[*] ~/Documents/Spark/Scala/supersimple/out/artifacts/supersimple_jar/supersimple.jar

这会导致以下错误

java.lang.ClassNotFoundException: SimpleScalaSpark
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.spark.util.Utils$.classForName(Utils.scala:229)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:695)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

我对于我所缺少的东西感到茫然…特别是考虑到它在IDE中按预期运行.

最佳答案 根据您的描述

,您没有提供正确的类名,因此无法找到该类.

只需用SimpleScalaSpark替换SimpleSparkScala即可

尝试运行此命令：

./spark-submit –class SimpleScalaSpark –master local [*]〜/ Documents / Spark / Scala / supersimple / out / artifacts / supersimple_jar / supersimple.jar

点赞