Airflow is the tool data teams use to run pipelines on a schedule: "every morning at 6am, pull yesterday's sales, clean them, load them into the warehouse, then refresh the dashboard". It's free, open source, and the orchestrator you'll see most often in data engineering job descriptions.
<aside> 🧭
Who this page is for: anyone who has never used Airflow. By the end you'll have Airflow running on your laptop and your first pipeline turning green in the UI, in about 30 minutes. Then the concepts that matter once you build real pipelines. Already running Airflow? Skip to Concepts worth understanding.
</aside>
<aside> 🔗
Before you start: this page assumes you've done Dev Environment Setup: a terminal you're comfortable opening, VS Code, and uv installed. If uv --version doesn't work yet, start there.
</aside>
A data pipeline is usually several steps that must happen in order: get the data, clean it, load it somewhere. Airflow's job is to run those steps in the right order, at the right time, retry them when they fail, and show you what happened.
Airflow is an orchestrator, not a processing engine. It tells other tools when to do the heavy work (your warehouse, dbt, Spark, a Python script) and keeps track of the results. If you find yourself loading millions of rows into memory inside Airflow, that's a sign the work belongs somewhere else.
Cost: running Airflow yourself is free (Apache 2.0 licence). You only pay if a company hosts it for you, such as Astronomer, Amazon MWAA or Google Cloud Composer. You don't need any of those to learn.
| Term | What it means |
|---|---|
| DAG | One pipeline. Stands for Directed Acyclic Graph: steps connected by arrows that only point forwards, so there are no loops. You write each DAG as a Python file |
| Task | One step in a DAG, such as "download the file" |
| Dependency | The order between tasks: extract must finish before transform starts |
| Schedule | When the DAG runs: daily, hourly, a custom time, or only when you trigger it |
| DAG run | One execution of a DAG, e.g. "Tuesday's run" |
| UI | Airflow's web page, where you watch runs, read logs and trigger DAGs |
Tested on Airflow 3.3.2 with Python 3.13 and uv 0.12 on macOS (October 2026). Airflow is updated regularly; check the latest version on PyPI and swap the version number in if it has moved on.
<aside> ⚠️
Windows users: Airflow does not run natively on Windows. Use WSL2, which gives you a real Linux system inside Windows. Open PowerShell as Administrator, run wsl --install, restart, then open the Ubuntu app from the Start menu. Do everything on this page inside that Ubuntu terminal, including installing uv with the macOS and Linux command from Dev Environment Setup. Microsoft's guide: Install WSL
</aside>
uv init my-airflow-project --no-package --python 3.13
cd my-airflow-project
The same flow as Dev Environment Setup. Airflow 3.3 needs Python 3.10 or newer, so --python 3.13 matters here: without it, uv may pick up an older Python already on your machine (macOS ships 3.9), and the install will fail.
You can delete the main.py file uv creates. Airflow doesn't use it.
Airflow keeps its settings, database, logs and your DAGs in a folder called Airflow home. By default that's ~/airflow, shared by every project on your computer. Keep it inside this project instead:
export AIRFLOW_HOME="$PWD/airflow"
export AIRFLOW__CORE__LOAD_EXAMPLES=False
export AIRFLOW__API__HOST=127.0.0.1