(
mw_dataset.withColumn("partition_id", sf.spark_partition_id())
.groupBy("partition_id")
.agg(sf.count(sf.col("partition_id")).alias("partition_count"))
.orderBy(sf.desc(sf.col("partition_count")))
.show()
)
Preview:
downloadDownload PNG
downloadDownload JPEG
downloadDownload SVG
Tip: You can change the style, width & colours of the snippet with the inspect tool before clicking Download!
Click to optimize width for Twitter