Promptfoo
A command-line tool for testing prompts, evaluating model output and red-teaming LLM applications and agents.
These buttons open the developer's own site, repository or store listing in a new tab. wares.gg does not host downloads.
1 more ways to get Promptfoo
Package managers
- Homebrew
brew install promptfoo
About Promptfoo
Promptfoo runs automated tests against prompts, models and complete LLM applications from the command line, started with npx. You describe test cases and checks, and it compares outputs across models. The documentation covers evaluating RAG pipelines, checking factuality and reducing hallucinations.
Its red-teaming mode generates attacks for a specific target, such as direct and indirect prompt injections, jailbreaks, data and PII leaks, business rule violations and insecure tool use in agents. The open-source tool sits alongside a commercial platform with enterprise features such as remediation reports.
Strengths
- Runs from the command line with a single npx command
- Compares prompts and models side by side
- Red-teaming covers prompt injection, jailbreaks and PII leaks
- Guides for RAG and factuality evaluation
Limitations
- Red-teaming features are intended for authorized testing of your own applications
- Some reporting features belong to the paid enterprise edition
Details
- Pricing
- FreemiumThe open-source CLI is free; enterprise features are sold separately.
- License
- Open source, license not stated
- Developer
- Promptfoo
- Platforms
- Command line
- How it runs
- Downloadable app
- Best suited for
- Developers who want repeatable tests and security checks for LLM apps
- Categories
- CLI tools, Security tools, AI developer tools
- Last verified
- Added
- Sources
Alternatives to Promptfoo
Compare allSoftware that can replace Promptfoo for an important use case, and what changes if you switch.
DeepEval
An open-source Python framework for unit-testing and evaluating the outputs of large language model applications.
DeepEval is a free open-source Python framework that treats LLM output checks like unit tests, replacing Promptfoo's npx command line workflow with Python code, and it does not advertise red-teaming.
Evidently
An open-source framework for evaluating, testing and monitoring LLM applications, RAG systems and ML models.
Evidently is an Apache 2.0 Python framework that adds drift, hallucination and data leak checks and covers classic ML models, but it needs Python code in place of Promptfoo's CLI.
Opik
An open-source platform from Comet for debugging, evaluating and monitoring LLM applications and agents.
Opik from Comet is a free open-source platform with a web interface that adds tracing and dashboards alongside evaluations, though self-hosting means running several services yourself.
Langfuse
An open-source platform for tracing, evaluating and managing prompts for LLM applications and AI agents.
Langfuse is an open-source web platform combining tracing, prompt management and evaluation, which requires instrumenting your application with its SDK rather than running tests from a command line.
Arize Phoenix
A self-hostable tool for tracing, evaluating and experimenting with LLM applications and agents.
Arize Phoenix offers tracing, evals, prompts and experiments in one self-hostable tool, under a source-available license instead of an open-source one, with some features in the paid Arize AX product.
MLflow
An open-source platform for tracing, evaluating and managing LLM applications, agents and machine learning models.
MLflow is an Apache 2.0 platform with 50+ built-in evaluation metrics, LLM judges and prompt versioning, and it also covers classic model training, though its broad feature set takes time to learn.
LangSmith
A hosted platform for tracing, monitoring and evaluating LLM applications and AI agents.
LangSmith is a proprietary hosted service focused on production agent tracing with online evals and alerts, replacing Promptfoo's local command-line testing, and self-hosting it requires Kubernetes.
Promptfoo as an alternative
Listings that name Promptfoo as an alternative.
lm-evaluation-harness
EleutherAI's command-line framework for few-shot evaluation of language models on many benchmark tasks.
Promptfoo compares prompts and models side by side from one npx command and adds red-teaming, but it tests your application rather than running hundreds of standard benchmarks.
Similar software
Related functionality, not necessarily a direct replacement.
Helicone
An open-source AI gateway and observability platform for routing, logging and debugging LLM requests.
Kiln
A free desktop workbench for building, evaluating and fine-tuning AI systems, with an open-source library.
Label Studio
An open-source data labeling tool for building training datasets and reviewing AI output.
