Spark 2.3.1
使用技巧
Spark-SQL
读取JSON
文件时反射表头
case class StudentInfo(id:Long,name:String,age:Int)
val example = spark.read.json("/data/result.json").as(StudentInfo)
example.show()
动态定义schema
在需要根据不同数据定义不同schema
val schemaInfo = "name age"
val fields = schemaInfo.map(item=> item.split(" ")
.map(item=>StructField(item,StringType,nullable=true))
val schema = StructType(fields)
val rowRDD = peopleRDD.map(_.split(" ").map(attributes=>Row(attributes(0),attributes(1))
val peopleDF = spark.createDataFrame(rowRDD,schema)
peopleDF.show()
Spark 2.3.1 on YARN
spark-submit
限制参数未生效
因为在spark-submit
时配置的executor-memory 2g
等没有生效,后来问同事说他也碰到这样的问题,解决方案就是动态的分配executor
,官方文档,中文文档
--conf spark.yarn.maxAppAttempts=1 --conf spark.dynamicAllocation.minExecutors=2 --conf spark.dynamicAllocation.maxExecutors=4 --conf spark.dynamicAllocation.initialExecutors=4