Alternatives to DeepEval

An open-source Python framework for unit-testing and evaluating the outputs of large language model applications. The listings below can replace it for an important use case. Each note says what changes if you switch.

Replacements

Listings that take over the same core job as DeepEval.

  • Promptfoo

    A command-line tool for testing prompts, evaluating model output and red-teaming LLM applications and agents.

    FreemiumOpen sourceCommand line

    Promptfoo runs tests from the command line with a single npx command, compares prompts and models side by side, and adds red-teaming, with some reporting in a paid edition.

  • Evidently

    An open-source framework for evaluating, testing and monitoring LLM applications, RAG systems and ML models.

    FreeOpen sourceCommand line

    Evidently is an Apache-2.0 Python framework that also covers classic ML models and checks for drift, hallucinations and data leaks alongside LLM evaluations.

  • Opik

    An open-source platform from Comet for debugging, evaluating and monitoring LLM applications and agents.

    FreeOpen sourceWebSelf-hosted

    Opik moves evaluation into a self-hostable platform with tracing and dashboards, which means running several services instead of a Python test library.

  • MLflow

    An open-source platform for tracing, evaluating and managing LLM applications, agents and machine learning models.

    FreeOpen sourceWebSelf-hostedCommand line

    MLflow brings 50+ built-in evaluation metrics and LLM judges inside a wider platform for tracing and model management, which needs a hosted server for team use.

  • Langfuse

    An open-source platform for tracing, evaluating and managing prompts for LLM applications and AI agents.

    FreemiumOpen sourceWebSelf-hosted

    Langfuse pairs evaluations with tracing, prompt management and annotation queues in a hosted or self-hosted platform, requiring you to instrument your application with its SDK.

  • Arize Phoenix

    A self-hostable tool for tracing, evaluating and experimenting with LLM applications and agents.

    FreemiumProprietaryWebSelf-hosted

    Arize Phoenix combines evals with tracing and experiments in a self-hostable tool, but uses a source-available license rather than an OSI-approved one.

Also worth comparing

These listings name DeepEval as their own alternative, so the relationship runs both ways.

  • lm-evaluation-harness

    EleutherAI's command-line framework for few-shot evaluation of language models on many benchmark tasks.

    FreeOpen sourceWindowsmacOSLinuxCommand line

    DeepEval treats LLM output checks as Python unit tests for applications instead of academic benchmark suites, and judge-based evaluations may need API access to a model.

Similar software

Related functionality, not a direct replacement.