Compare software

Promptfoo vs lm-evaluation-harness: catalog facts
Promptfoo	lm-evaluation-harness
Freemium The open-source CLI is free; enterprise features are sold separately.	Free Free and open source.
Command line	Windows, macOS, Linux, Command line
Runs from the command line with a single npx command Compares prompts and models side by side	Hundreds of benchmark tasks in one tool New tasks defined through YAML templates
Red-teaming features are intended for authorized testing of your own applications Some reporting features belong to the paid enterprise edition	Command-line and Python only, no graphical interface Running large benchmarks needs substantial compute
Downloadable app	Downloadable app
Open source	Open source
License not stated	License: MIT
Not stated if an account is needed	No account needed
Not stated if it works offline	Works offline
Official website Full listing and sources Checked September 26, 2026	Official website Full listing and sources Checked September 26, 2026

Catalog facts only. Anything “not stated” is unconfirmed. Check full listings for details.