TabbyAPI

A lightweight, OpenAI-compatible API server for running ExLlama-format language models on your own hardware.

FreeOpen sourceWindowsLinuxSelf-hostedCommand line
Official websitegithub.com

These buttons open the developer's own site, repository or store listing in a new tab. wares.gg does not host downloads.

About TabbyAPI

TabbyAPI is the official API server for the ExLlama inference backend. It loads ExLlama-format models on your own GPU and exposes them through an OpenAI-compatible API. Chat front ends and other tools written for that API can then use the local model.

Configuration lives in a YAML file, and optional API tokens control access. The project supports LoRAs and sampler overrides, and it ships start scripts for Windows and Linux plus Docker files. It suits people running quantised models locally who want a fast backend.

Strengths

  • OpenAI-compatible endpoints for existing clients
  • Lightweight server focused on ExLlama models
  • LoRA support and sampler overrides
  • Docker and start scripts included

Limitations

  • Only serves ExLlama-format models
  • No built-in chat interface; pair it with a separate front end

Details

Pricing
FreeFree and open source.
License
Open source, license not stated
Developer
theroyallab
Platforms
Windows, Linux, Self-hosted, Command line
How it runs
Downloadable app, Self-hosted
Account
Not required
Works offline
Yes
Best suited for
Running ExLlama-quantised models behind an OpenAI-style API
Last verified
Added
Provenance
Facts checked against the developer's own pages and store listings, 1 sources on file.

Alternatives to TabbyAPI

Compare all

Software that can replace TabbyAPI for an important use case, and what changes if you switch.

  • llama.cpp

    The C++ engine most local AI apps are built on, running language models on ordinary hardware.

    FreeOpen sourceWindowsmacOSLinuxAndroidCommand line

    llama.cpp runs GGUF-style quantised models as one binary with a built-in server and web interface, supporting many backends instead of ExLlama format only.

  • KoboldCpp

    A single executable that runs GGUF language models locally, with a web interface and no installation at all.

    FreeOpen sourceWindowsmacOSLinux

    KoboldCpp is a single executable for GGUF models that exposes OpenAI, Ollama, A1111 and ComfyUI compatible endpoints and includes its own web interface.

  • TextGen

    Chat with local language models using the TextGen desktop interface.

    FreeOpen sourceWindowsmacOSLinux

    TextGen is an AGPL-3.0 desktop interface that includes API access alongside a built-in chat interface, so no separate front end is required.

  • Ollama

    The simplest way to pull down an open language model and run it on your own machine, from one command or a desktop app.

    FreemiumOpen sourceWindowsmacOSLinuxCommand line

    Ollama downloads and runs models with one command and serves a localhost REST API, removing ExLlama format limits with a desktop app option.

  • LocalAI

    Serve language, image and speech models on your own hardware.

    FreeOpen sourceLinuxmacOSSelf-hosted

    LocalAI is MIT-licensed and serves language, image and speech models with compatible APIs for common services, broader than TabbyAPI's language-model focus.

  • vLLM

    High-throughput, memory-efficient inference and serving engine for LLMs.

    FreeOpen sourceLinuxCommand line

    vLLM is an Apache-2.0 high-throughput serving engine for Linux, suited to production-scale deployments on appropriate GPU hardware rather than lightweight single-user use.

  • Xinference

    An open-source inference server for running language, speech and multimodal models through one API.

    FreeOpen sourcemacOSLinuxSelf-hostedCommand line

    Xinference serves language, speech and multimodal models behind one GPT-compatible API on macOS, Linux or servers, but has no Windows listing.

  • SGLang

    A serving framework for running large language models and multimodal models on your own GPUs.

    FreeOpen sourceSelf-hostedCommand line

    SGLang is a high-throughput serving framework for language and multimodal models on your own GPU servers, heavier than TabbyAPI's lightweight setup.

Similar software

Related functionality, not necessarily a direct replacement.

Report a wrong fact or a dead link on this listing