Preview:
(
    mw_dataset.withColumn("partition_id", sf.spark_partition_id())
              .groupBy("partition_id")
              .agg(sf.count(sf.col("partition_id")).alias("partition_count"))
              .orderBy(sf.desc(sf.col("partition_count")))
              .show()
)
downloadDownload PNG downloadDownload JPEG downloadDownload SVG

Tip: You can change the style, width & colours of the snippet with the inspect tool before clicking Download!

Click to optimize width for Twitter