A partition is a fragment of a DataFrame processed by a single task on an executor. Their number determines the level of parallelism: too few leaves cores idle, too many creates tiny tasks and scheduling overhead. Their size and balance strongly influence performance.
Example
df.rdd.getNumPartitions()