Как перенести строку в столбец с помощью Spark-SQL?

Question

я следующие данные в моей таблице t1

col1 | col2 | 
sess-1 | read | 
sess-1 | meet | 
sess-1 | walk | 
sess-2 | watch | 
sess-2 | sleep | 
sess-2 | run | 
sess-2 | drive | 

ожидается выход:

col1 | col2     | 
sess-1 | read,meet,walk  | 
sess-2 | watch,sleep,run,drive | 

Я использую искру 1.4.0

Answer 1

Проверить искру

aggregateByKey

scala> val babyNamesCSV = sc.parallelize(List(("David", 6), ("Abby", 4), ("David", 5), ("Abby", 5))) 
babyNamesCSV: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[0] at parallelize at <console>:12 


scala> babyNamesCSV.aggregateByKey(0)((k,v) => v.toInt+k, (v,k) => k+v).collect 
res1: Array[(String, Int)] = Array((Abby,9), (David,11)) 

вышеописанный пример может помочь понять

или Агрегирование https://spark.apache.org/docs/0.6.0/api/core/spark/Aggregator.html

Answer 2

// create RDD data 
scala> val data = sc.parallelize(List(("sess-1","read"), ("sess-1","meet"), 
    ("sess-1","walk"), ("sess-2","watch"),("sess-2","sleep"), 
    ("sess-2","run"),("sess-2","drive"))) 

//groupByKey will return Iterable[String] CompactBuffer** 
scala> val dataCB = data.groupByKey()` 

//map CompactBuffer to List 
scala> val tx = dataCB.map{case (col1,col2) => (col1,col2.toList)}.collect 

data: org.apache.spark.rdd.RDD[(String, String)] = 
ParallelCollectionRDD[211] at parallelize at <console>:26 

dataCB: org.apache.spark.rdd.RDD[(String, Iterable[String])] = 
ShuffledRDD[212] at groupByKey at <console>:30 

tx: Array[(String, List[String])] = Array((sess-1,List(read, meet, 
walk)), (sess-2,List(watch, sleep, run, drive))) 

//groupByKey and map to List can also achieved in one statment 
scala> val dataCB = data.groupByKey().map{case (col1,col2) 
    => (col1,col2.toList)}.collect