Orchestrating Data Pipelines with Apache Airflow
Orchestrating Pipelines with Airflow is an advanced-level course of 5 lessons (about 1 h 40 min), with 17 exercises and 15 quiz questions. Prerequisites: Transforming Data with dbt, Intermediate Python (Functions, Modules, OOP). The course needs a full project environment, planned for a later phase of the platform.
- Data engineering
- Advanced
- 5 lessons
- 1 h 40 min
- 17 exercises
- Full project environment (phase 2)
The course in brief
This course teaches you how to turn standalone scripts into reliable pipelines orchestrated by Apache Airflow 3. You write DAGs with TaskFlow, master logical dates, catchup, and backfill, make your tasks idempotent and fault-tolerant, wait for external data with sensors and assets, then test, monitor, and deploy your DAGs. The pure Python exercises reproduce the scheduler's mechanisms so you understand them in depth.
What you will be able to do
- Write an Airflow 3 DAG with TaskFlow and operators, and predict its execution order
- Schedule a DAG and use the logical date, data interval, catchup, and backfill
- Make a pipeline robust with retries, timeouts, idempotence, and trigger rules
- Wait for external data with sensors and chain DAGs through assets
- Test, monitor, and deploy DAGs while handling connections and secrets correctly
Syllabus: 5 lessons
- Explain what an orchestrator brings compared with a simple cron job
- Write an Airflow 3 DAG with the TaskFlow API and classic operators
- Declare dependencies and predict the order in which tasks run
DAG (directed acyclic graph)TaskFlow APIOperatorsTask dependenciesXCom
- 02
Scheduling, Logical Dates, and Catchup
- Configure a DAG's schedule with a cron expression or a preset
- Distinguish the logical date, the data interval, and the actual run time
- Master catchup and backfill to reprocess a past period
- Use template variables to target the right partition
Scheduling (cron, presets)Logical dateData intervalCatchup and backfillJinja templates
- 03
Retries, Idempotence, and Trigger Rules
- Configure automatic retries and timeouts for a task
- Design idempotent tasks that can be rerun without side effects
- Choose a suitable trigger rule (cleanup, alert, optional branch)
Automatic retriesIdempotenceTrigger rulesFailure callbacksExecution timeouts
- 04
Sensors and Dependencies Between Pipelines
- Wait for an external condition with a well-configured sensor
- Choose between poke, reschedule, and deferrable modes
- Chain DAGs through data using assets
SensorsPoke and reschedule modesDeferrable operatorsAssets and data-driven scheduling
- 05
Monitoring, Testing, and Deployment
- Test DAGs before putting them into production
- Monitor runs and diagnose a failure from the logs
- Describe Airflow 3's architecture and choose a deployment mode
- Manage connections, variables, and secrets without writing them in the code
DAG testsMonitoring and alertsArchitecture and executorsConnections, variables, and secretsDeployment
Final project
Orchestrated and robust daily sales pipeline
Design an Airflow 3 DAG that waits for a fictional partner's daily sales file, cleans it, loads it idempotently into a table partitioned by day, then publishes an asset that triggers a second DAG computing indicators. The pipeline must support a one-month backfill without duplicates, alert on final failure, and pass integrity tests in continuous integration.
Deliverables
- The Git repository with the two DAG files and the transformations module
- The pytest tests: DAG integrity and transformation logic
- A demonstration of a one-month backfill, with the row count before and after a second run
- The deployment file (Docker Compose or Helm values) and connection management with no secrets in the code
- A short operations note: sensor timeouts, retry policy, alerts
Assessment criteria
- Tasks use the logical date and are idempotent (no duplicates after a rerun)
- Retries, timeouts, and sensors are tuned in a justified way (timeout, mode, backoff)
- Dependencies between DAGs go through assets
- Integrity and logic tests pass in continuous integration
- No credentials appear in the code; the deployment is reproducible
Frequently asked questions
Does this course cover Airflow 2 or Airflow 3?
Airflow 3. The differences from Airflow 2 that matter are pointed out: imports from airflow.sdk, assets (formerly datasets), catchup disabled by default, trigger-based cron scheduling, and the new backfill command.
Do I need to install Airflow to follow the course?
The code exercises run in pure Python in the browser and reproduce the scheduler's mechanisms. The hands-on tasks are done in an Airflow environment, for example the one provided by the Docker Compose file in the official documentation.
What is the difference between Airflow and cron?
cron runs commands at fixed times. Airflow also handles dependencies between tasks, retries, run history, catching up on past periods, waiting for external data, and a monitoring interface.
Does Airflow process the data itself?
No. Airflow orchestrates: it triggers and monitors tasks that rely on other tools (SQL, Spark, dbt, Python scripts). Large volumes of data should not flow through Airflow.
This course, told for your job
Say what you want to be able to do: “Orchestrating Pipelines with Airflow” takes its place in your path, with examples from your field. The concepts stay those of the course.