Partitioned writes

Partitioned writing saves a DataFrame by creating a subfolder per column value (partitionBy), often in Parquet. Reads that filter on this column scan only the relevant folders. Choose a column of moderate cardinality to avoid a multitude of small files.

Example

df.write.partitionBy('annee', 'mois').mode('overwrite').parquet('sortie/ventes')

Related concepts

The whole glossary