Podemos apagar o pyspark dataframe usando python
df.rdd.saveAsPickleFile(filename)
Relieved Rook
df.rdd.saveAsPickleFile(filename)