Alternatives to lm-evaluation-harness
EleutherAI's command-line framework for few-shot evaluation of language models on many benchmark tasks. The listings below can replace it for an important use case. Each note says what changes if you switch.
The original
lm-evaluation-harness
EleutherAI's command-line framework for few-shot evaluation of language models on many benchmark tasks.
Replacements
Listings that take over the same core job as lm-evaluation-harness.
Promptfoo
A command-line tool for testing prompts, evaluating model output and red-teaming LLM applications and agents.
Promptfoo compares prompts and models side by side from one npx command and adds red-teaming, but it tests your application rather than running hundreds of standard benchmarks.
DeepEval
An open-source Python framework for unit-testing and evaluating the outputs of large language model applications.
DeepEval treats LLM output checks as Python unit tests for applications instead of academic benchmark suites, and judge-based evaluations may need API access to a model.
Kiln
A free desktop workbench for building, evaluating and fine-tuning AI systems, with an open-source library.
Kiln moves evaluation into a free desktop app with fine-tuning and synthetic data, replacing the command line with a graphical interface, though the app's license is not stated.
Similar software
Related functionality, not a direct replacement.
MLflow
An open-source platform for tracing, evaluating and managing LLM applications, agents and machine learning models.
LlamaFactory
Web UI and command-line toolkit for fine-tuning over 100 open language and vision-language models.
Hugging Face Hub
A web platform for hosting, finding and sharing machine learning models, datasets and demo apps.
Opik
An open-source platform from Comet for debugging, evaluating and monitoring LLM applications and agents.
Langfuse
An open-source platform for tracing, evaluating and managing prompts for LLM applications and AI agents.
