Apache Airflow

A self-hosted platform for writing, scheduling and monitoring workflows defined as Python code.

FreeOpen sourceWebSelf-hosted
Official websiteairflow.apache.orgSource codegithub.comDocumentationairflow.apache.org

These buttons open the developer's own site, repository or store listing in a new tab. wares.gg does not host downloads.

About Apache Airflow

Apache Airflow lets you define pipelines in plain Python, using ordinary language features such as loops and date handling to generate tasks dynamically. Parameters are handled with Jinja templating, and a library of ready-made operators connects to many outside systems.

A web interface shows the status and logs of running and finished tasks and lets you schedule and manage workflows. The modular design uses a message queue to spread work across many workers. It suits data engineers and developers who want scheduled jobs written as code.

Strengths

  • Workflows are plain Python, so tasks can be generated dynamically
  • Web UI with status and logs for every task run
  • Scales out across many workers through a message queue
  • Large set of ready-made operators for common integrations

Limitations

  • Requires Python knowledge to define workflows
  • You run and maintain the scheduler, workers and web server yourself

Details

Pricing
FreeFree and open source, developed as an Apache Software Foundation project.
License
Apache-2.0
Developer
The Apache Airflow contributors
Platforms
Web, Self-hosted
How it runs
Self-hosted
Best suited for
Data teams scheduling and monitoring pipelines written in Python
Last verified
Added
Provenance
Facts checked against the developer's own pages and store listings, 1 sources on file.

Alternatives to Apache Airflow

Compare all

Software that can replace Apache Airflow for an important use case, and what changes if you switch.

  • Prefect

    A workflow orchestration tool for scheduling, retrying and observing Python data and ML pipelines.

    FreemiumOpen sourceWebSelf-hostedCommand line

    Prefect defines flows with plain Python decorators and adds built-in retries, with a paid hosted platform holding some features alongside the Apache-2.0 open-source core.

  • Dagster

    An open-source orchestrator for building, scheduling and monitoring data pipelines, with a hosted Dagster+ option.

    FreemiumOpen sourceWebSelf-hosted

    Dagster uses an asset-based model with lineage and freshness tracking and offers a hosted Dagster+ option, though its announced merger with Prefect makes the roadmap uncertain.

  • Luigi

    A Python tool from Spotify for building, scheduling and visualising pipelines of batch jobs.

    FreeOpen sourceWindowsmacOSLinuxCommand line

    Luigi is Spotify's lighter Python tool for chaining batch jobs with automatic dependency resolution and Hadoop support, aimed at batch jobs rather than event-driven workflows.

  • Flyte

    An open-source orchestration platform for durable data, machine learning and AI agent workflows written in Python.

    FreeOpen sourceWebSelf-hostedCommand line

    Flyte orchestrates data, machine learning and AI agent workflows in plain Python with failure recovery, but Flyte 1 users need to migrate to Flyte 2.

  • Kestra

    An open-source declarative orchestration platform for scheduled and event-driven workflows, self-hosted or in the cloud.

    FreemiumOpen sourceWebSelf-hosted

    Kestra defines workflows declaratively rather than in Python, offers more than 2,000 plugins and event-driven triggers, and can run self-hosted or in the cloud.

  • Dagu

    A self-hosted workflow orchestrator that runs scripts, SSH commands and containers from declarative YAML files.

    FreeOpen sourceSelf-hostedCommand line

    Dagu runs workflows from YAML files as a single binary with no database, suiting smaller teams, but workflows are not defined in Python.

  • Windmill

    An open-source, self-hostable platform for turning scripts into workflows, internal apps and scheduled jobs.

    FreemiumOpen sourceWebSelf-hosted

    Windmill turns scripts in TypeScript, Python, Go, Bash and other languages into workflows, internal apps and scheduled jobs, with some features in an enterprise edition.

Apache Airflow as an alternative

Listings that name Apache Airflow as an alternative.

  • Apache NiFi

    A visual dataflow system for automating the movement and transformation of data between systems.

    FreeOpen sourceWebSelf-hosted

    Apache Airflow defines pipelines as Python code instead of a visual flow designer, and focuses on scheduling and monitoring tasks rather than runtime flow changes.

  • Conductor

    An open-source, event-driven workflow engine for durable execution of application and AI agent workflows.

    FreeOpen sourceSelf-hosted

    Apache Airflow schedules workflows written as Python code with a web UI for task status and logs, scaling across workers, though you maintain the scheduler, workers and web server yourself.

  • Cronicle

    A self-hosted, distributed task scheduler and runner with a web interface.

    FreeOpen sourceWebSelf-hosted

    Apache Airflow schedules and monitors workflows defined as Python code across many workers, which requires Python knowledge and running the scheduler, workers and web server yourself.

Similar software

Related functionality, not necessarily a direct replacement.

Report a wrong fact or a dead link on this listing