Alternatives to Luigi
A Python tool from Spotify for building, scheduling and visualising pipelines of batch jobs. The listings below can replace it for an important use case. Each note says what changes if you switch.
The original
Luigi
A Python tool from Spotify for building, scheduling and visualising pipelines of batch jobs.
Replacements
Listings that take over the same core job as Luigi.
Apache Airflow
A self-hosted platform for writing, scheduling and monitoring workflows defined as Python code.
Apache Airflow also defines workflows in Python but adds a web UI with logs per task and scales across workers, requiring you to run a scheduler and web server.
Prefect
A workflow orchestration tool for scheduling, retrying and observing Python data and ML pipelines.
Prefect uses plain Python decorators with built-in retries, scheduling and a run history dashboard, and offers a paid hosted platform for some features.
Dagster
An open-source orchestrator for building, scheduling and monitoring data pipelines, with a hosted Dagster+ option.
Dagster uses an asset-based model with lineage and freshness tracking and works with dbt, with an uncertain roadmap after its announced merger with Prefect.
Flyte
An open-source orchestration platform for durable data, machine learning and AI agent workflows written in Python.
Flyte orchestrates Python data and ML workflows with recovery from failures, though Flyte 1 users need to migrate to Flyte 2.
Kestra
An open-source declarative orchestration platform for scheduled and event-driven workflows, self-hosted or in the cloud.
Kestra replaces Python code with declarative workflows, more than 2,000 plugins and event-driven triggers, but is heavier than a simple batch tool.
Apache NiFi
A visual dataflow system for automating the movement and transformation of data between systems.
Apache NiFi moves data through a browser-based visual flow designer with provenance tracking instead of Python code, aimed at server deployments.
Similar software
Related functionality, not a direct replacement.
Dagu
A self-hosted workflow orchestrator that runs scripts, SSH commands and containers from declarative YAML files.
Windmill
An open-source, self-hostable platform for turning scripts into workflows, internal apps and scheduled jobs.
GNU Parallel
A command-line utility that runs shell commands and scripts in parallel across CPU cores or machines.
Conductor
An open-source, event-driven workflow engine for durable execution of application and AI agent workflows.
