Skew and salting

Data skew appears when one key holds a large share of the rows: one partition becomes much heavier than the others and slows down the whole stage. Salting mitigates the problem by adding a random suffix to the key to spread the load, then recombining the partial results.

Example

df.withColumn('cle_salee', F.concat('cle', F.lit('_'), (F.rand() * 10).cast('int')))

Related concepts

The whole glossary