Caching

Caching keeps a DataFrame in memory (and optionally on disk) after its first computation, to reuse it without recomputing it from the source. It is useful when the same result serves several actions. Since it is lazy, an action is needed to fill it, and unpersist() frees the space.

Example

df.cache()
df.count()  # fills the cache

Related concepts

The whole glossary