SparkSession and Spark DataFrames

When pandas is no longer enough

pandas loads all the data into the memory of a single machine. Beyond a few tens of gigabytes, or when a job must read thousands of files from a data lake, you need to distribute the work. Apache Spark is the most widely used distributed computing engine for this: it splits data into partitions and processes them in parallel across several machines, in memory as much as possible. You use it from Python with PySpark. This course relies on the DataFrame API of Spark 3.5 and 4.x.

Spark is not always the right answer: for a few million rows, pandas or DuckDB on one machine is simpler and often faster. Spark becomes relevant when volume or parallelism demands it.

The architecture in three roles

• The driver runs your script, builds the execution plan, and distributes the work.
• The executors are processes launched on the cluster’s machines; each one processes tasks, one task corresponding to one data partition.
• The cluster manager (Kubernetes, YARN, Spark standalone mode, or a managed service) allocates resources.

Locally, everything runs in a single process (master("local[*]") uses all cores): convenient for learning, with exactly the same API.

Creating a session and reading data

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = (SparkSession.builder
         .appName("ventes")
         .master("local[*]")          # on a cluster, this setting is provided at launch
         .getOrCreate())

schema = "id INT, jour DATE, region STRING, produit STRING, montant DOUBLE"

ventes = (spark.read
          .option("header", True)
          .option("mode", "DROPMALFORMED")   # ignore unreadable rows
          .schema(schema)
          .csv("s3a://lake/brut/ventes/"))   # a whole folder: all its files

ventes.printSchema()
ventes.show(5)

The explicit schema avoids two problems of inferSchema=True: an extra pass over the data to guess the types, and wrongly guessed types (a postal code read as an integer loses its leading zero). The read mode decides what happens to malformed rows: PERMISSIVE (the default) keeps them, setting to null whatever cannot be read; DROPMALFORMED discards them; FAILFAST stops the read at the first error.

For data that is already prepared, prefer Parquet: a columnar, compressed format that carries its own schema (spark.read.parquet(...)).

Transforming a DataFrame

The API resembles pandas, but with a declarative, SQL-inspired style:

resultat = (ventes
    .filter(F.col("montant") > 0)
    .withColumn("ttc", F.round(F.col("montant") * 1.2, 2))
    .withColumn("gamme", F.when(F.col("montant") >= 200, "haut").otherwise("standard"))
    .select("jour", "region", "produit", "ttc", "gamme"))

Three rules:

1. A DataFrame is immutable: withColumn does not modify ventes, it returns a new DataFrame. So you must reassign or chain.
2. Column expressions (F.col, F.when, F.to_date, F.upper…) run inside Spark’s engine, optimized and distributed. A Python function applied row by row (UDF) is much slower: reserve it for cases with no native equivalent.
3. Nothing has been computed yet at this point: Spark waits for an action (show, count, write…), as we will see in the next lesson.

Bringing data back to the driver

show(n) displays a few rows; count() counts; collect() brings all the rows back into the driver’s memory. On a real volume, collect() crashes the driver: prefer limit(n), show(), or write the result (write.parquet). To pass an already aggregated, small result to pandas, toPandas() is fine.

Key takeaways

• Driver = conductor, executors = workers, one task per partition.
• Explicit schema and a chosen read mode; Parquet for prepared data.
• Immutable DataFrame, F functions rather than row-by-row Python, and never collect() on a large volume.