apache/airflow

What it does

Apache Airflow lets you write workflows as code in Python and orchestrate their tasks according to declared dependencies. The scheduler runs tasks across a pool of workers while the web UI shows pipeline state, lets you retry or backfill runs, and inspects generated code. Workflows are commonly called DAGs, and assets record data dependencies between them.

The model fits workflows whose structure is mostly static and changes slowly, which keeps task boundaries and reruns predictable. Airflow also orchestrates ML training, evaluation, and deployment pipelines, and can coordinate the steps of LLM-based pipelines without being the agent itself. It is not a streaming system; real-time work is handled by pulling batches from streams. Tasks are expected to be idempotent and should not carry large payloads between them, and XCom is meant for metadata rather than bulk data.

Quick start

The library and application are both shipped as one PyPI package, so a plain pip install apache-airflow can resolve to unusable dependency combinations. Use a published constraints file for your Airflow version and Python version. This example follows the current stable 3.3.2 tag and Python 3.10:

pip install 'apache-airflow==3.3.2' \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.3.2/constraints-3.10.txt"

Extras such as database or cloud integrations are installed in the same way:

pip install 'apache-airflow[postgres,google]==3.3.2' \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.3.2/constraints-3.10.txt"

After installation, the official getting-started guide covers initializing the database, starting the scheduler and web server, and running a first DAG; the same documentation contains the full tutorial. Provider packages are installed separately. Docker images, Helm charts, and GitHub tags are convenience distributions built from official release sources.

Runtime and compatibility

Airflow runs on POSIX systems; production use is supported only on Linux distributions, with the community Docker image based on Debian Bookworm. Windows is not natively supported and is used through WSL2 or Linux containers. SQLite is fine for tests and local development but should not be used in production; PostgreSQL and MySQL are the tested production databases, and MariaDB is not tested or recommended.

The stable 3.3.2 release supports Python 3.10 through 3.14 on AMD64 and ARM64 and Kubernetes 1.30 through 1.35. The 2.x line has reached end of life: its final release is 2.11.2, with end of life on April 22, 2026. Version support follows Python and Kubernetes end-of-life dates, so pick a release that matches your platform and upgrade before support ends.

Dependency management is deliberately loose to let users upgrade libraries used by DAGs, with constraints files pinning a known-working set. Core dependencies such as SQLAlchemy, Alembic, Flask, Werkzeug, Celery, and the Kubernetes client are upper-bounded individually. Only pip and uv installations are officially supported; Poetry and pip-tools require converting the constraint files yourself. The project is licensed under Apache-2.0.

HotRepo's take

The repository's on-site record is thin: one Python daily snapshot from November 6, 2026 placed it 5th on that day's Python list with 16 stars gained, against totals of 47,389 stars and 18,317 forks. With no cross-day snapshot series or discussion threads available, that single entry only confirms continued attention on a heavily used project; it does not establish a growth trend. The more useful signal is the documented scope: a long-lived Apache project with a broad provider ecosystem and a detailed compatibility and release policy, aimed at teams that need reproducible, code-defined batch pipelines rather than a lightweight script runner.