Apache Airflow

📦 PyPI: https://pypi.org/project/apache-airflow/3.3.0/

📚 Docs: https://airflow.apache.org/docs/apache-airflow/3.3.0/

🛠 Release Notes: https://airflow.apache.org/docs/apache-airflow/3.3.0/release_notes.html

🐳 Docker Image: "docker pull apache/airflow:3.3.0"

🚏 Constraints: https://github.com/apache/airflow/tree/constraints-3.3.0

Significant Changes

Asset Partitioning (#64571, #65447, #66030, #66848, #67184, #67475, #67716, #68978)

Building on the asset partitioning introduced in 3.2.0, Airflow 3.3.0 substantially expands how a

single upstream asset event fans out to partitioned downstream Dag runs. New partition mappers —

`RollupMapper (many-to-one), FanOutMapper (one-to-many), and FixedKeyMapper` +

`SegmentWindow` (categorical rollup) — compose with time windows (day/week/month/quarter/year) and a

`wait_policy (WaitForAll or MinimumCount(n)`) to control when partitioned runs fire. Windows

can fan out forward or backward in time, and total fan-out per upstream event is bounded by the new

`[scheduler] partition_mapper_max_downstream_keys` config (configurable per mapper). Airflow 3.3.0

also adds the `PartitionedAtRuntime` timetable, which lets a Dag declare that its partition key(s)

are assigned when the run starts rather than mapped from an upstream event.

For detailed usage instructions, see :doc:/authoring-and-scheduling/assets.

Task and Asset State Store (#65759, #66073, #66160, #66463, #66586, #66859, #67041, #67292, #67319)

Airflow 3.3.0 introduces a first-class state store for tasks and assets (AIP-103). Tasks can persist

arbitrary key-value state that survives across retries and runs via a new `task_state_store` accessor, and

assets can carry their own state via `asset_state_store` — both available from the Task SDK. State is kept

in the metadata database by default, or in a custom worker-side backend (``[workers]

state_store_backend``), supports per-key retention with periodic garbage collection and an optional

`clear_on_success`, and is fully manageable through the Core API and Execution API.

For detailed usage instructions, see :doc:/core-concepts/task-and-asset-state-store.

Pluggable Retry Policies (#65474)

Task retry behaviour is now pluggable (AIP-105). In addition to a fixed `retries` count, you can

attach a custom retry policy that decides whether and when a task is retried, enabling strategies such

as retrying only on specific exceptions or backing off based on custom logic.

For detailed usage instructions, see :ref:concepts:retry-policies.

Language Task SDK (Java and Go) (#65958, #67161, #67635, #67699)

Airflow 3.3.0 adds a Coordinator layer (AIP-108) that lets individual task implementations be written

in non-Python languages while the Dag and its scheduling stay in Python. A task is declared in the Dag

with `@task.stub(queue=...)`; the worker routes it to a configured coordinator

(`JavaCoordinator for JVM languages, ExecutableCoordinator` for self-contained native binaries

such as Go) that runs the task in a language runtime and proxies Variables, Connections, and XComs back

through the Execution API.

.. warning::

The Coordinator layer and the Java/Go SDKs are experimental in 3.3.0 and may change in future

versions based on user feedback.

For detailed usage instructions, see :doc:/authoring-and-scheduling/language-sdks/index.

Dag bundle version on clear, rerun, and backfill (#63884)

The new `rerun_with_latest_version` setting controls whether a cleared, rerun, or backfilled Dag run

uses the latest bundle version or the original version from the initial run. The default is resolved

by precedence: an explicit request parameter/CLI flag, then the Dag-level `rerun_with_latest_version`,

then `[core] rerun_with_latest_version, and finally False for clear/rerun and True` for

backfills (preserving historical behaviour). Airflow 2.x

<!--

Licensed to the Apache Software Foundation (ASF) under one

or more contributor license agreements. See the NOTICE file

distributed with this work for additional information

regarding copyright ownership. The ASF licenses this file

to you under the Apache License, Version 2.0 (the

"License"); you may not use this file except in compliance

with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,

software distributed under the License is distributed on an

"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY

KIND, either express or implied. See the License for the

specific language governing permissions and limitations

under the License.

-->

<!-- START Apache Airflow, please keep comment here to allow auto update of PyPI readme.md -->

Apache Airflow

| Category | Badges |

|------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

| License | License |

| PyPI | PyPI version PyPI - Python Version PyPI - Downloads |

| Containers | Docker Pulls Docker Stars Artifact HUB |

| Community | Contributors Slack Status Commit Activity LFX Health Score |

| Dev tools | prek |

| Version | Build Status |

|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|

| Main | Tests AMD main Tests ARM main |

| 3.x | Tests AMD 3.2 Tests ARM 3.2 |

<picture width="500">

<img

src="https://github.com/apache/airflow/blob/19ebcac2395ef9a6b6ded3a2faa29dc960c1e635/docs/apache-airflow/img/logos/wordmark_1.png?raw=true"

alt="Apache Airflow logo"

/>

</picture>

Apache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows.

When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative.

Use Airflow to author workflows (Dags) that orchestrate tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on Dags a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.

<!-- END Apache Airflow, please keep comment here to allow auto update of PyPI readme.md -->

<!-- START doctoc generated TOC please keep comment here to allow auto update -->

<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->

Table of contents

• Project Focus
• Principles
• Requirements
• Getting started
• Installing from PyPI
• Installation
• Official source code
• Convenience packages
• User Interface
• Semantic versioning
• Version Life Cycle
• Support for Python and Kubernetes versions
• Base OS support for reference Airflow images
• Approach to dependencies of Airflow
• Contributing
• Community standards
• Agent-assisted contribution (apache-magpie)
• Voting Policy
• Who uses Apache Airflow?
• Who maintains Apache Airflow?
• What goes into the next release?
• Can I use the Apache Airflow logo in my presentation?
• Links
• Sponsors

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

Project Focus

Airflow works best with workflows that are mostly static and slowly changing. When the Dag structure is similar from one run to the next, it clarifies the unit of work and continuity. Other similar projects include Luigi, Oozie and Azkaban.

Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent (i.e., results of the task will be the same, and will not create duplicated data in a destination system), and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's XCom feature). For high-volume, data-intensive tasks, a best practice is to delegate to external services specializing in that type of work.

Airflow is not a streaming solution, but it is often used to process real-time data, pulling data off streams in batches.

Principles

• Dynamic: Pipelines are defined in code, enabling dynamic dag generation and parameterization.
• Extensible: The Airflow framework includes a wide range of built-in operators and can be extended to fit your needs.
• Flexible: Airflow leverages the Jinja templating engine, allowing rich customizations.

<!-- START Requirements, please keep comment here to allow auto update of PyPI readme.md -->

Requirements

Apache Airflow is tested with:

| | Main version (dev) | Stable version (3.3.0) | Deprecate version (2.11.2) |

|------------|------------------------------------|-------------------------------------|------------------------------|

| Python | 3.10, 3.11, 3.12, 3.13, 3.14 | 3.10, 3.11, 3.12, 3.13, 3.14 | 3.10, 3.11, 3.12 |

| Platform | AMD64/ARM64 | AMD64/ARM64 | AMD64/ARM64(\*) |

| Kubernetes | 1.30, 1.31, 1.32, 1.33, 1.34, 1.35 | 1.30, 1.31, 1.32, 1.33, 1.34, 1.35 | 1.26, 1.27, 1.28, 1.29, 1.30 |

| PostgreSQL | 14, 15, 16, 17, 18 | 14, 15, 16, 17, 18 | 12, 13, 14, 15, 16 |

| MySQL | 8.0, 8.4, Innovation | 8.0, 8.4, Innovation | 8.0, Innovation |

| SQLite | 3.15.0+ | 3.15.0+ | 3.15.0+ |

\* Experimental

Note: MariaDB is not tested/recommended.

Note: SQLite is used in Airflow tests. Do not use it in production. We recommend

using the latest stable version of SQLite for local development.

Note: Airflow currently can be run on POSIX-compliant Operating Systems. For development, it is regularly

tested on fairly modern Linux Distros and recent versions of macOS.

On Windows you can run it via WSL2 (Windows Subsystem for Linux 2) or via Linux Containers.

The work to add Windows support is tracked via #10388, but

it is not a high priority. You should only use Linux-based distros as "Production" execution environment

as this is the only environment that is supported. The only distro that is used in our CI tests and that

is used in the Community managed DockerHub image is

Debian Bookworm.

<!-- END Requirements, please keep comment here to allow auto update of PyPI readme.md -->

<!-- START Getting started, please keep comment here to allow auto update of PyPI readme.md -->

Getting started

Visit the official Airflow website documentation (latest stable release) for help with

installing Airflow,

getting started, or walking

through a more complete tutorial.

Note: If you're looking for documentation for the main branch (latest development branch): you can find it on s.apache.org/airflow-docs.

For more information on Airflow Improvement Proposals (AIPs), visit

the Airflow Wiki.

Documentation for dependent projects like provider distributions, Docker image, Helm Chart, you'll find it in the documentation index.

<!-- END Getting started, please keep comment here to allow auto update of PyPI readme.md -->

<!-- START Installing from PyPI, please keep comment here to allow auto update of PyPI readme.md -->

Installing from PyPI

We publish Apache Airflow as apache-airflow package in PyPI. Installing it however might be sometimes tricky

because Airflow is a bit of both a library and application. Libraries usually keep their dependencies open, and

applications usually pin them, but we should do neither and both simultaneously. We decided to keep

our dependencies as open as possible (in pyproject.toml) so users can install different versions of libraries

if needed. This means that pip install apache-airflow will not work from time to time or will

produce unusable Airflow installation.

To have repeatable installation, however, we keep a set of "known-to-be-working" constraint

files in the orphan constraints-main and constraints-2-0 branches. We keep those "known-to-be-working"

constraints files separately per major/minor Python version.

You can use them as constraint files when installing Airflow from PyPI. Note that you have to specify

correct Airflow tag/version/branch and Python versions in the URL.

1. Installing just Airflow:

Note: Only pip installation is currently officiall