Luigi

A Python tool from Spotify for building, scheduling and visualising pipelines of batch jobs.

FreeOpen sourceWindowsmacOSLinuxCommand line
Official websitegithub.com

These buttons open the developer's own site, repository or store listing in a new tab. wares.gg does not host downloads.

About Luigi

Luigi is used to chain batch jobs into pipelines. Each task is written in Python and declares what it depends on, and Luigi works out the order to run them in, manages the workflow and shows the pipeline visually. It also has built-in Hadoop support.

It is maintained on GitHub by Spotify and supports current Python 3 releases from 3.10 up. It suits data engineers who want dependency handling for scheduled batch processing without adopting a larger orchestration platform.

Strengths

  • Automatic dependency resolution between tasks
  • Built-in visualisation of pipelines
  • Hadoop support included
  • Supports current Python 3 releases

Limitations

  • Pipelines must be written in Python
  • Aimed at batch jobs rather than event-driven workflows

Details

Pricing
FreeFree and open source, developed by Spotify.
License
Open source, license not stated
Developer
Spotify
Platforms
Windows, macOS, Linux, Command line
How it runs
Downloadable app
Account
Not required
Works offline
Yes
Best suited for
Data engineers chaining Python batch jobs into dependable pipelines
Last verified
Added
Provenance
Facts checked against the developer's own pages and store listings, 1 sources on file.

Alternatives to Luigi

Compare all

Software that can replace Luigi for an important use case, and what changes if you switch.

  • Apache Airflow

    A self-hosted platform for writing, scheduling and monitoring workflows defined as Python code.

    FreeOpen sourceWebSelf-hosted

    Apache Airflow also defines workflows in Python but adds a web UI with logs per task and scales across workers, requiring you to run a scheduler and web server.

  • Prefect

    A workflow orchestration tool for scheduling, retrying and observing Python data and ML pipelines.

    FreemiumOpen sourceWebSelf-hostedCommand line

    Prefect uses plain Python decorators with built-in retries, scheduling and a run history dashboard, and offers a paid hosted platform for some features.

  • Dagster

    An open-source orchestrator for building, scheduling and monitoring data pipelines, with a hosted Dagster+ option.

    FreemiumOpen sourceWebSelf-hosted

    Dagster uses an asset-based model with lineage and freshness tracking and works with dbt, with an uncertain roadmap after its announced merger with Prefect.

  • Flyte

    An open-source orchestration platform for durable data, machine learning and AI agent workflows written in Python.

    FreeOpen sourceWebSelf-hostedCommand line

    Flyte orchestrates Python data and ML workflows with recovery from failures, though Flyte 1 users need to migrate to Flyte 2.

  • Kestra

    An open-source declarative orchestration platform for scheduled and event-driven workflows, self-hosted or in the cloud.

    FreemiumOpen sourceWebSelf-hosted

    Kestra replaces Python code with declarative workflows, more than 2,000 plugins and event-driven triggers, but is heavier than a simple batch tool.

  • Apache NiFi

    A visual dataflow system for automating the movement and transformation of data between systems.

    FreeOpen sourceWebSelf-hosted

    Apache NiFi moves data through a browser-based visual flow designer with provenance tracking instead of Python code, aimed at server deployments.

Similar software

Related functionality, not necessarily a direct replacement.

Report a wrong fact or a dead link on this listing