spark stream

2023年1月11日 266次阅读 来源: Hystrix_Hu

Dstream 是一个 rdd的队列。
当spark stream 窗口函数的间隔不是batchDuration的倍数时会报错。

Exception in thread "main" java.lang.Exception: The window duration of windowed DStream (10000 ms) must be a multiple of the slide duration of parent DStream (3000 ms)
   at org.apache.spark.streaming.dstream.WindowedDStream.<init>(WindowedDStream.scala:35)
   at org.apache.spark.streaming.dstream.DStream$$anonfun$window$1.apply(DStream.scala:766)
   at org.apache.spark.streaming.dstream.DStream$$anonfun$window$1.apply(DStream.scala:766)
   at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
   at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
   at org.apache.spark.SparkContext.withScope(SparkContext.scala:679)
   at org.apache.spark.streaming.StreamingContext.withScope(StreamingContext.scala:264)
   at org.apache.spark.streaming.dstream.DStream.window(DStream.scala:765)
    原文作者：Hystrix_Hu
    原文地址: https://www.jianshu.com/p/f828efc985a3
    本文转自网络文章，转载此文章仅为分享知识，如有侵权，请联系博主进行删除。
点赞