DAGs, Tasks, and Dependencies

Why an orchestrator?

A real data pipeline chains several steps: extract files or an API, clean them, load them into a warehouse, then refresh reporting tables. One cron line per script works at first, then quickly shows its limits: how do you start the load only if the extraction succeeded? How do you automatically rerun a step that failed because of an unstable network? How do you reprocess the last three days? Where do you see the run history?

Apache Airflow answers these questions. It does not transform data itself: it orchestrates tasks (Python scripts, SQL queries, Spark jobs, shell commands), in the right order, at the right time, with retries, logs, and a monitoring interface. This course uses Airflow 3; the important differences from Airflow 2 are pointed out.

The vocabulary: DAG, task, run

• A DAG (Directed Acyclic Graph) describes a pipeline: tasks linked by dependencies. “Acyclic” means you can never return to a task you have already passed: A → B → A is forbidden.
• A task is a unit of work. It is created from an operator (BashOperator, SQLExecuteQueryOperator…) or from a function decorated with @task.
• A DAG run is an instance of the pipeline for a given date; each task in it becomes a task instance with its own state (queued, running, success, failed, up_for_retry, skipped…).

A DAG’s Python file is a declaration: Airflow re-reads it regularly to learn the structure of the graph. So make no network calls and do no heavy computation at module level: that code would be executed every time the file is parsed.

A first DAG with TaskFlow

The TaskFlow API turns Python functions into tasks:

import pendulum
from airflow.sdk import dag, task


@dag(
    dag_id="ventes_quotidiennes",
    schedule="@daily",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
    tags=["ventes"],
)
def ventes_quotidiennes():

    @task
    def extraire() -> str:
        # Downloads today's export and returns its path (a small value)
        return "/data/brut/ventes.csv"

    @task
    def transformer(chemin: str) -> str:
        # Cleans the file and writes a clean version
        return "/data/propre/ventes.parquet"

    @task
    def charger(chemin: str) -> None:
        print(f"Loading {chemin} into the warehouse")

    charger(transformer(extraire()))


ventes_quotidiennes()

The call charger(transformer(extraire())) does not run anything: it builds the graph extraire → transformer → charger. Return values are passed through XCom (cross-communication), a small store kept in Airflow’s metadata database. Pass paths, IDs, or counters through it, never a DataFrame with millions of rows: write the data to storage (file, database, object storage) and pass its location.

Classic operators and explicit dependencies

The same pipeline can mix operators and TaskFlow:

from airflow.sdk import DAG
from airflow.providers.standard.operators.bash import BashOperator
from airflow.providers.standard.operators.empty import EmptyOperator

with DAG(dag_id="entrepot_nuit", schedule="0 2 * * *",
         start_date=pendulum.datetime(2026, 1, 1, tz="UTC")):
    debut = EmptyOperator(task_id="debut")
    extraire = BashOperator(task_id="extraire", bash_command="python extract.py")
    charger_dw = BashOperator(task_id="charger_dw", bash_command="python load_dw.py")
    charger_lac = BashOperator(task_id="charger_lac", bash_command="python load_lake.py")
    rapport = EmptyOperator(task_id="rapport")

    debut >> extraire >> [charger_dw, charger_lac] >> rapport

a >> b reads as “b runs after a”. A list creates a fan-out: charger_dw and charger_lac both depend on extraire and can run in parallel; rapport waits for both. In Airflow 3, the basic operators (BashOperator, PythonOperator, EmptyOperator, file sensors) ship in the standard provider, hence the airflow.providers.standard import.

How Airflow chooses the order

The scheduler applies a simple principle: a task becomes runnable when all its upstream tasks have finished according to its trigger rule (by default all_success). Among the ready tasks, several can be started at the same time, within the limits of resources (slots, pools, max_active_tasks). This order is a topological sort of the graph: it exists only because the graph has no cycle. That is what you will program in the exercises.

Key takeaways

• Airflow orchestrates: it triggers, orders, retries, and traces; the processing happens in the tasks.
• A DAG has no cycle; the file that describes it must stay lightweight.
• >> and TaskFlow value passing create the dependencies; XCom carries only small values.