如何在Jupyter上的HDInsight Spark集群上提交python wordcount

2023年3月18日 267次阅读

我试图在Spark HDInsight集群上运行
python wordcount,我正在从Jupyter运行它.我不确定这是否是正确的方法,但我找不到任何有关如何在HDInsight Spark集群上提交独立python应用程序的帮助.

代码 ：

import pyspark
import operator
from pyspark import SparkConf
from pyspark import SparkContext
import atexit
from operator import add
conf = SparkConf().setMaster("yarn-client").setAppName("WC")
sc = SparkContext(conf = conf)
atexit.register(lambda: sc.stop())

input = sc.textFile("wasb:///example/data/gutenberg/davinci.txt")
words = input.flatMap(lambda x: x.split())
wordCount = words.map(lambda x: (str(x),1)).reduceByKey(add)

wordCount.saveAsTextFile("wasb:///example/outputspark")

并且我得到并且不明白的错误消息：

ValueError                                Traceback (most recent call last)
<ipython-input-2-8a9d4f2cb5e8> in <module>()
      6 from operator import add
      7 import atexit
----> 8 sc = SparkContext('yarn-client')
      9 
     10 input = sc.textFile("wasb:///example/data/gutenberg/davinci.txt")

/usr/hdp/current/spark-client/python/pyspark/context.pyc in __init__(self, master, appName, sparkHome, pyFiles, environment, batchSize, serializer, conf, gateway, jsc, profiler_cls)
    108         """
    109         self._callsite = first_spark_call() or CallSite(None, None, None)
--> 110         SparkContext._ensure_initialized(self, gateway=gateway)
    111         try:
    112             self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,

/usr/hdp/current/spark-client/python/pyspark/context.pyc in _ensure_initialized(cls, instance, gateway)
    248                         " created by %s at %s:%s "
    249                         % (currentAppName, currentMaster,
--> 250                             callsite.function, callsite.file, callsite.linenum))
    251                 else:
    252                     SparkContext._active_spark_context = instance

ValueError: Cannot run multiple SparkContexts at once; existing SparkContext(app=pyspark-shell, master=yarn-client) created by __init__ at <ipython-input-1-86beedbc8a46>:7

实际上可以用这种方式运行python作业吗？如果是 – 似乎是SparkContext定义的问题……我尝试了不同的方法：

sc = SparkContext('spark://headnodehost:7077', 'pyspark')

和

conf = SparkConf().setMaster("yarn-client").setAppName("WordCount1")
sc = SparkContext(conf = conf)

但没有成功.什么是运行作业或配置SparkContext的正确方法？

最佳答案 如果从Jupyter笔记本运行,则为您预先创建Spark上下文,创建单独的上下文将不正确.要解决此问题,只需删除创建上下文的行并直接从以下位置开始：

input = sc.textFile("wasb:///example/data/gutenberg/davinci.txt")

如果需要运行独立程序,可以使用pyspark从命令行运行它,或者使用在集群上运行的Livy服务器使用REST API提交它.

点赞