Spark实例-通过HDFS文件实时统计

通过Spark Streaming,实时监控HDFS目录,发现有文件时,实时进行计算。

package com.spark.streaming

import org.apache.spark.SparkConf
import org.apache.spark.streaming.StreamingContext
import org.apache.spark.streaming.Seconds

/*
 *
 * 通过HDFS文件实时统计
 */
object HDFSWordCount extends App {
  val conf = new SparkConf()
  .setAppName("HDFSWordCount")
  //.setMaster("hdfs://hadoop01:9000/")
  val ssc = new StreamingContext(conf,Seconds(5))

  val lines = ssc.textFileStream("hdfs://hadoop01:9000/wordcount_dir")
  val words = lines.flatMap(line=>line.split(" "))
  val paris = words.map((_,1))
  val wordCount=paris.reduceByKey(_+_)
  wordCount.print()
  ssc.start()
  ssc.awaitTermination()
  ssc.stop()
}
    原文作者:不圆的石头
    原文地址: https://www.jianshu.com/p/9325fcedc0f9
    本文转自网络文章,转载此文章仅为分享知识,如有侵权,请联系博主进行删除。
点赞