JustVugg/colibri

What it does

colibri is an inference engine for large mixture-of-experts language models on ordinary machines. It keeps the dense parts of a model in RAM and leaves routed experts on disk, reading them when the router selects them. A per-layer cache with a learned hot set decides which experts stay in RAM or VRAM, so the placement affects speed but not the router's decisions or weight precision. It is written in pure C with one engine file per model family, and runs on CPU, Vulkan, CUDA and Metal.

The engines cover GLM-5.2/5.3, GLM-5.3-Flash, Inkling, Kimi K3, DeepSeek V4 Flash and V4.1 Flash, MiMo-V2.6, Qwen3.6/Qwen3.8 and their sibling models, OLMoE, plus Qwen-Image-2.1 for image generation and three decision models. Models can come from the project's int4 containers or from existing GGUF files. coli serve exposes OpenAI- and Anthropic-compatible HTTP APIs, so coding CLIs and SDKs can connect to a local model. A dashboard shows routing, speed and per-turn profiling, and a /v1/systemone endpoint returns probabilities for constrained choices instead of generating text.

The intended audience is people who want to run models far larger than their RAM or VRAM would normally allow, and researchers who want a small engine they can measure and modify.

Quick start

Minimum requirements are 8 GB of RAM, 22 GB free disk for the smallest model, and an internet connection; a GPU is optional. The one-step setup is available for Windows, Linux and macOS and is documented in the quick start guide.

On Ubuntu or Debian:

sudo apt install git python3 build-essential
git clone https://github.com/JustVugg/colibri
cd colibri
./start-here.sh

On macOS, install the Xcode command-line tools and use Homebrew for libomp, git and Python, then run the same ./start-here.sh. On Windows, download the repository ZIP, unzip it and run START-HERE.bat. The setup inspects RAM, disk, CPU and GPUs, recommends a model that fits, builds or downloads the engine, downloads the model with resume support, then starts the server and prints the dashboard and API addresses:

Browser:             http://127.0.0.1:8000/
OpenAI base URL:     http://127.0.0.1:8000/v1
Anthropic base URL:  http://127.0.0.1:8000

To skip the questionnaire, pass --yes; to choose the model explicitly, use --model ID; to store models elsewhere, use --dir DIR. --list shows every model against the current machine. For a manual install, download a release for your platform or build from c/ with ./setup.sh, obtain a model, then run ./coli chat --model PATH, ./coli web --model PATH or ./coli serve --model PATH (install by hand).

Requirements and limits

Model size matters: the smallest option in the catalog is a 23 GB download needing about 10 GB of RAM to start, while large streaming models are tens to hundreds of GB and rely on fast NVMe storage for throughput. The setup table gives separate minimum and comfortable RAM figures, and recommends the most capable model that fits in RAM before listing disk-streamed alternatives.

Backend support is uneven. Vulkan works on any GPU with a Vulkan 1.2 driver; CUDA is built on Linux when the toolkit is present and ships as a separate DLL on Windows; Metal is available in the macOS release and from source with METAL=1, but the one-step setup builds for CPU. The engine's environment variables such as COLI_VULKAN=1 or COLI_METAL=1 toggle GPU use, and integrated-GPU acceleration is only enabled by default for selected Qwen engines where measurements showed a gain.

Most chat engines support documents and tool calling, but not all do: Inkling and OLMoE lack tool calling. Image input is limited to specific models. coli serve --kv-slots supports up to 16 concurrent conversations.

The code is Apache-2.0, while each model keeps its own license; Qwen-Image-2.1 weights are non-commercial only. On WSL, keeping the model directory on the Linux disk rather than /mnt/c avoids a large slowdown.

HotRepo's take

colibri's recorded rise is notable for an inference engine: on the all-language daily list it went from about 27.3k stars on 2026-09-10 to a best rank of 10 on 2026-11-02, with single-day gains of 2,233 and 2,035 stars on 2026-09-14 and 2026-09-15, and it passed 46k stars by the latest snapshot. The README's benchmark tables are unusually concrete, reporting hardware classes and per-machine speeds such as 0.05–0.1 tok/s on a 25 GB laptop for GLM-5.2 and 104.8 tok/s on an RTX PRO 4000 for a 35B model, which makes the project's claims checkable rather than purely promotional. The main caveat visible in the material is operational: disk-streamed inference is sensitive to storage speed, and the largest models are only usable at low token rates on consumer hardware.