Alternatives to llama.cpp
The C++ engine most local AI apps are built on, running language models on ordinary hardware. The listings below can replace it for an important use case. Each note says what changes if you switch.
The original
llama.cpp
The C++ engine most local AI apps are built on, running language models on ordinary hardware.
Replacements
Listings that take over the same core job as llama.cpp.
Ollama
The simplest way to pull down an open language model and run it on your own machine, from one command or a desktop app.
Ollama wraps local model running in one command or a desktop app with a REST API and handles model downloads, trading llama.cpp's low-level control for simplicity.
vLLM
High-throughput, memory-efficient inference and serving engine for LLMs.
vLLM is an Apache-2.0 high-throughput serving engine for production deployments, but it is Linux-only and needs suitable GPU hardware for good performance.
KoboldCpp
A single executable that runs GGUF language models locally, with a web interface and no installation at all.
KoboldCpp is a single AGPL-licensed executable that runs GGUF models and adds image generation, speech and vision with several compatible API endpoints.
llamafile
Packages a language model and the code to run it into one executable file that works on several operating systems without installation.
llamafile bundles the model and runtime into one Apache-licensed file that runs on Windows, macOS, Linux and BSD, though files become large.
LocalAI
Serve language, image and speech models on your own hardware.
LocalAI serves language, image and speech models with compatible APIs under MIT, targeting Linux, macOS and self-hosting rather than Windows or Android.
LM Studio
A polished desktop app for downloading, running and chatting with local language models, with an OpenAI-compatible server built in.
LM Studio provides a polished desktop interface with Hugging Face search, an OpenAI-compatible server and MLX support, but it is closed source unlike MIT-licensed llama.cpp.
TextGen
Chat with local language models using the TextGen desktop interface.
TextGen offers a desktop chat interface for local models with vision and API access in portable builds, under AGPL-3.0 rather than MIT.
Similar software
Related functionality, not a direct replacement.
Jan
An open-source ChatGPT replacement that runs entirely offline, and can also front cloud providers when you want them.
GPT4All
A local AI chat app with document chat built in, from Nomic AI. Stable, MIT licensed, but not updated since early 2025.
Open WebUI
A self-hosted chat interface that talks to Ollama and any OpenAI-compatible API, so your local models get a proper front end.
Chatbox
A desktop and mobile chat client for AI models, where you supply your own API keys or point it at a local model.
SillyTavern
A locally-installed front end for language models aimed at long-form roleplay and character-driven writing.
Whisper
An open-source speech recognition model from OpenAI for transcribing and translating audio on your own machine.
