Consolidation - RowNumber

PHOTO EMBED

Wed Jun 09 2021 09:44:02 GMT+0000 (UTC)

Saved by @Safrugans ##pyspark

windowSpec = W\
  .partitionBy(F.col("pos"))\
  .orderBy(
    F.col("pos").desc(), 
    F.col("op_ts").desc(), 
    F.col("current_ts").desc()
  	)

df = df\
  .withColumn(
  	"row_num", 
  	F.row_number().over(windowSpec)
	)\
  .filter(F.col("row_num") > '1')
content_copyCOPY