Alternatives to vLLM

High-throughput, memory-efficient inference and serving engine for LLMs. The listings below can replace it for an important use case. Each note says what changes if you switch.

Replacements

Listings that take over the same core job as vLLM.

  • llama.cpp

    The C++ engine most local AI apps are built on, running language models on ordinary hardware.

    FreeOpen sourceWindowsmacOSLinuxAndroidCommand line

    llama.cpp is an MIT C++ engine that runs on Windows, macOS, Linux and Android with CPU and GPU hybrid inference and heavy quantisation, suiting ordinary hardware over high-throughput serving.

  • Ollama

    The simplest way to pull down an open language model and run it on your own machine, from one command or a desktop app.

    FreemiumOpen sourceWindowsmacOSLinuxCommand line

    Ollama is an MIT tool that downloads and runs models with one command and a localhost REST API, aimed at individual users rather than production-scale serving.

  • LocalAI

    Serve language, image and speech models on your own hardware.

    FreeOpen sourceLinuxmacOSSelf-hosted

    LocalAI is an MIT server for language, image and speech models on Linux and macOS, offering compatible APIs for several common AI services.

  • LM Studio

    A polished desktop app for downloading, running and chatting with local language models, with an OpenAI-compatible server built in.

    FreemiumProprietaryWindowsmacOSLinux

    LM Studio is a closed-source desktop app that serves local models on OpenAI-compatible endpoints over llama.cpp and MLX, aimed at trying models rather than production serving.

  • KoboldCpp

    A single executable that runs GGUF language models locally, with a web interface and no installation at all.

    FreeOpen sourceWindowsmacOSLinux

    KoboldCpp is a single AGPL-3.0 executable that runs GGUF models on CPU or GPU and exposes OpenAI and Ollama compatible endpoints, needing no installation.

  • SGLang

    A serving framework for running large language models and multimodal models on your own GPUs.

    FreeOpen sourceSelf-hostedCommand line

    SGLang is also a high-throughput serving framework for GPU servers, and it handles multimodal models as well as language models, with Docker files and benchmark tooling.

  • Xinference

    An open-source inference server for running language, speech and multimodal models through one API.

    FreeOpen sourcemacOSLinuxSelf-hostedCommand line

    Xinference serves language, speech and multimodal models behind one GPT-compatible API, running on a laptop or scaling to servers on macOS and Linux.

  • TabbyAPI

    A lightweight, OpenAI-compatible API server for running ExLlama-format language models on your own hardware.

    FreeOpen sourceWindowsLinuxSelf-hostedCommand line

    TabbyAPI is a lightweight OpenAI-compatible server limited to ExLlama-format models, and it runs on Windows as well as Linux.

Also worth comparing

These listings name vLLM as their own alternative, so the relationship runs both ways.

  • GPUStack

    An open-source GPU cluster manager for serving AI models with vLLM and SGLang on your own hardware.

    FreeOpen sourceSelf-hosted

    vLLM is an Apache-2.0 high-throughput serving engine on Linux, which GPUStack builds on, without GPUStack's cluster pooling and SSH-accessible instances.

Similar software

Related functionality, not a direct replacement.