Lazy evaluation

Lazy evaluation means Spark computes nothing when you declare transformations: it builds a logical plan. The computation starts only when an action is called. This lets Spark optimize the whole plan, group operations together, and avoid reading unnecessary data.

Example

df2 = df.filter('montant > 0')  # nothing is computed
df2.count()  # the action triggers the computation

Related concepts

The whole glossary