mostlygeek/llama-swap

llama-swap is worth a try if you run several local models on one machine and are tired of juggling ports, launch scripts and clients. It is a Go binary plus a YAML config that listens on one address and routes each model request to the right backend process, starting it only when needed.

The problem

A developer with one GPU wants to use a chat model, an embedding model and an image or speech model from the same client. Normally each backend has to be started by hand on its own port, the client has to be reconfigured per model, and an idle process may hold VRAM the next model needs. With llama-swap, the client keeps talking to one OpenAI- or Anthropic-style endpoint, the proxy matches the requested model to a configured command and spawns it, then returns the response. The result is a single stable address that presents many models, with switching handled outside the client.

What's different

Rather than bundling an inference engine, llama-swap treats each model as a command described in the config. That makes it agnostic to the backend: the README lists llama.cpp and its forks, vLLM, stable-diffusion.cpp, audio.cpp, ComfyUI and whisper.cpp, and says the proxy works with any OpenAI/Anthropic-compatible server. Upgrading or swapping an inference server does not mean changing the routing layer.

The proxy exposes a broad set of endpoints rather than just chat: completions, responses, embeddings, audio speech and transcriptions, image generation and edits, Anthropic messages and token counting, plus llama.cpp-specific routes such as rerank and infill. A live web UI provides a playground, token metrics and request/response inspection, and models can be loaded and unloaded manually (README). The main trade-off is configuration: the value comes from describing each model's launch command correctly, and the tool is a supervisor, not a model manager, so it adds a process layer that text clients do not see.

Who should try it

A good fit is a developer with a single GPU box who wants one local endpoint for several models and several tools: an editor plugin, a script and a chat client can all point at the same host and switch models by name. It is less suitable for someone running exactly one model always loaded, where the proxy adds a hop and a config file for little benefit. Docker users have a clear path: the unified images bundle llama-server, ik-llama-server, stable-diffusion.cpp, whisper.cpp and audio.cpp for CUDA 12, CUDA 13 and Vulkan (container images), while the legacy image tracks llama.cpp's own server container but lacks image, speech and ik-llama-server support (legacy base). Homebrew, MacPorts, WinGet and release binaries cover non-container setups; MacPorts and WinGet are community-maintained rather than official parts of the project (WinGet issue).

Requirements and limitations

The central prerequisite is local inference hardware and at least one backend server the config can launch; llama-swap itself does not provide model weights or an engine. Because swapping is on-demand, the first request for a model pays its startup time, and only one model can occupy limited VRAM at a time in a typical setup. The unified Docker images are GPU-specific: CUDA 13 covers Ampere through Blackwell, the CUDA 12 image covers older cards that CUDA 13 dropped, and Vulkan serves AMD and other Vulkan-capable GPUs, so matching the image to the hardware matters (README).

Ongoing upkeep is mostly config maintenance: every new model or changed launch flag means editing the model list, and backend upgrades are your responsibility since the project deliberately does not pin an engine. The README notes some endpoint limitations, such as /props requiring a model query parameter and ignoring autoload.

HotRepo's take

The repository was created on 2024-10-04, and HotRepo's only tracked board observation is 2026-10-25, when it appeared on the daily trending board with 6,073 stars and +12 that day. That single snapshot is not enough to judge persistence either way. Before adopting, check the config examples for your exact backend: the project's value depends on whether your server's command line can be expressed cleanly and whether preload behaviour fits your workflow.