Evidently
An open-source framework for evaluating, testing and monitoring LLM applications, RAG systems and ML models.
These buttons open the developer's own site, repository or store listing in a new tab. wares.gg does not host downloads.
About Evidently
Evidently checks the quality of AI systems in one framework. It covers LLM outputs, RAG applications, AI agents and traditional ML models, looking for problems such as hallucinations, edge cases, leaks of sensitive data, risky outputs, jailbreaks and data drift.
It is released under the Apache 2.0 license and has been downloaded tens of millions of times. It suits teams that want to test AI behaviour on every update and keep watching it after release.
Strengths
- Covers both LLM systems and classic ML models
- Checks for drift, hallucinations and data leaks
- Apache 2.0 license
Limitations
- Requires writing Python code to set up checks
- Some evaluations depend on access to a judge model
Details
- Pricing
- FreeFree and open source under the Apache 2.0 license.
- License
- Apache-2.0
- Developer
- Evidently AI
- Platforms
- Command line
- How it runs
- Downloadable app
- Best suited for
- Teams testing and monitoring LLM and ML systems
- Categories
- AI developer tools
- Last verified
- Added
- Sources
Alternatives to Evidently
Compare allSoftware that can replace Evidently for an important use case, and what changes if you switch.
DeepEval
An open-source Python framework for unit-testing and evaluating the outputs of large language model applications.
DeepEval focuses on LLM outputs as unit tests in Python, without Evidently's coverage of classic ML models and drift monitoring.
MLflow
An open-source platform for tracing, evaluating and managing LLM applications, agents and machine learning models.
MLflow is also Apache-2.0 and covers both LLM apps and classic models, adding tracing, prompt versioning and experiment management, but a team setup needs a hosted server.
Opik
An open-source platform from Comet for debugging, evaluating and monitoring LLM applications and agents.
Opik offers LLM tracing, automated evaluations and dashboards in a self-hostable platform, but focuses on LLM apps and agents rather than classic ML models.
Promptfoo
A command-line tool for testing prompts, evaluating model output and red-teaming LLM applications and agents.
Promptfoo tests prompts and model output from the command line and adds red-teaming for prompt injection and PII leaks, but does not cover classic ML monitoring.
Arize Phoenix
A self-hostable tool for tracing, evaluating and experimenting with LLM applications and agents.
Arize Phoenix provides tracing, evals and experiments for LLM apps with a hosted option, under a source-available license instead of Apache 2.0.
Langfuse
An open-source platform for tracing, evaluating and managing prompts for LLM applications and AI agents.
Langfuse centres on tracing, prompt management and evals for LLM apps in a hosted or self-hosted platform, without classic ML drift checks.
Similar software
Related functionality, not necessarily a direct replacement.
Weights & Biases
A hosted platform for tracking machine learning experiments, evaluating models and tracing LLM applications.
TensorBoard
A visualization toolkit for inspecting machine learning training runs, metrics and model graphs.
LangSmith
A hosted platform for tracing, monitoring and evaluating LLM applications and AI agents.
lm-evaluation-harness
EleutherAI's command-line framework for few-shot evaluation of language models on many benchmark tasks.
