Distributed Data Processing with Apache Spark
Distributed Processing with Spark is an advanced-level course of 5 lessons (about 1 h 40 min), with 16 exercises and 15 quiz questions. Prerequisites: Data Analysis with pandas and NumPy. The course needs a full project environment, planned for a later phase of the platform.
- Data engineering
- Advanced
- 5 lessons
- 1 h 40 min
- 16 exercises
- Full project environment (phase 2)
The course in brief
This course teaches you how to move from pandas to distributed computing with Apache Spark. You understand the driver and executors architecture, work with PySpark DataFrames, master lazy evaluation, partitions, shuffle, and join strategies, write Spark SQL queries with window functions, then diagnose and speed up a job. The exercises reproduce the engine's internal mechanisms in pure Python and SQL.
What you will be able to do
- Read data with an explicit schema and transform PySpark DataFrames
- Explain lazy evaluation, jobs, stages, tasks, and the shuffle
- Control partitions and choose a suitable join strategy
- Write Spark SQL queries with aggregations and window functions
- Write partitioned tables idempotently
- Diagnose a slow job and fix skew, UDFs, and small files
Syllabus: 5 lessons
- Explain the role of the driver, the executors, and the cluster manager
- Create a SparkSession and read a file with an explicit schema
- Select, filter, and enrich a DataFrame with the functions of pyspark.sql.functions
Driver, executors, and clusterSparkSessionSpark DataFrameSchema and file readingColumn functions
- 02
Transformations, Actions, and Lazy Evaluation
- Distinguish transformations from actions
- Explain lazy evaluation and the role of the optimizer
- Relate jobs, stages, tasks, and the shuffle
- Know when to cache an intermediate result
Lazy evaluationTransformations and actionsNarrow and wide transformationsJobs, stages, and tasksCaching
- 03
Partitions, Shuffle, and Joins
- Control the number of partitions with repartition and coalesce
- Understand hash partitioning and two-phase aggregation
- Choose between a sort-merge join and a broadcast join
PartitionsHash partitioningrepartition and coalesceDistributed joinsBroadcast join
- 04
Spark SQL and Window Functions
- Query DataFrames in SQL with temporary views
- Move from SQL syntax to the DataFrame API and back
- Compute rankings and running totals with window functions
- Save partitioned tables idempotently
Spark SQLTemporary views and tablesWindow functionsPartitioned writes
- 05
Optimizing a Spark Job
- Diagnose a slow job with the Spark UI and explain()
- Apply the main levers: columnar formats, pruning, broadcast, partitions
- Detect and handle data skew
- Replace Python UDFs with native or vectorized functions
Job diagnosisAdaptive query execution (AQE)Skew and saltingPruning and columnar formatsUDFs and pandas UDFs
Final project
Spark pipeline for analyzing multi-store sales
From generated data (several million fictional sales, a reference table of stores and products, with one store that accounts for a large share of the sales), build a PySpark job that reads the raw data with an explicit schema, cleans it, enriches it, computes daily indicators and rankings by region, then writes a Parquet table partitioned by day idempotently. Measure and optimize the job.
Deliverables
- The PySpark script (or notebook) runnable in local mode, with the transformations factored into functions
- The Spark SQL ranking and running-total queries
- The Parquet table partitioned by day and proof that rerunning a single day creates no duplicates
- An optimization report: before/after plans (explain), stage durations, skew handling
Assessment criteria
- The results are correct and verified on a sample
- No needless collect or Python UDF; native functions are used
- Joins use the right strategy and keys are checked (uniqueness, orphans)
- The write is partitioned, idempotent, and free of a multitude of small files
- Optimizations are justified by measurements
Frequently asked questions
What is the difference between pandas and Spark?
pandas works in memory on a single machine; Spark splits data into partitions processed in parallel across several machines, with lazy evaluation and an optimizer. For modest volumes, pandas remains simpler.
Do I need a cluster to follow the course?
No. The code exercises run in pure Python and SQL in the browser. The hands-on tasks use PySpark in local mode, which offers the same API as a cluster.
Does the course use the RDD API?
The course relies on the DataFrame API and Spark SQL, which are recommended today. The RDD API appears only to illustrate the map/reduce pattern.
Which Spark versions are covered?
Spark 3.5 and 4.x. The concepts presented (lazy evaluation, shuffle, broadcast, AQE, window functions) are common to these versions.
This course, told for your job
Say what you want to be able to do: “Distributed Processing with Spark” takes its place in your path, with examples from your field. The concepts stay those of the course.