scala – Spark – 删除两个数组类型列之间的相交元素

2019年7月28日 274次阅读

我有这样的数据帧

+---------+--------------------+----------------------------+
|     Name|                rem1|        quota               |
+---------+--------------------+----------------------------+
|Customer_3|[258, 259, 260, 2...|[1, 2, 3, 4, 5, 6, 7,..500]|
|Customer_4|[18, 19, 20, 27, ...|[1, 2, 3, 4, 5, 6, 7,..500]|
|Customer_5|[16, 17, 51, 52, ...|[1, 2, 3, 4, 5, 6, 7,..500]|
|Customer_6|[6, 7, 8, 9, 10, ...|[1, 2, 3, 4, 5, 6, 7,..500]|
|Customer_7|[0, 30, 31, 32, 3...|[1, 2, 3, 4, 5, 6, 7,..500]|

我想从配额中删除rem1中的列表值并创建为一个新列.我试过了.

val dfleft = dfpci_remove2.withColumn("left",$"quota".filter($"rem1"))

<console>:123: error: value filter is not a member of org.apache.spark.sql.ColumnName

请指教.

最佳答案 您可以这样使用列中的过滤器,您可以编写如下的udf

  val filterList = udf((a: Seq[Int], b: Seq[Int]) => a diff b)

  df.withColumn("left", filterList($"rem1", $"quota") )

这应该给你预期的结果.

希望这可以帮助！

点赞