Removes log files from spark write

PHOTO EMBED

Tue Oct 26 2021 14:55:00 GMT+0000 (UTC)

Saved by @Safrugans ##pyspark

%sql 
SET spark.sql.sources.commitProtocolClass=org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol


spark.conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs","false")
content_copyCOPY