wici-ai/inferssd

InferSSD is an inference engine for large mixture-of-experts (MoE) language models that streams expert weights from SSD. Built on llama.cpp and ggml, it is aimed at people who want to run very large MoE checkpoints on a single machine whose memory cannot hold the whole model, using standard GGUF files rather than a custom format. It exposes a terminal chat and a local browser UI with an OpenAI-compatible endpoint.

What changes in practice

Instead of loading every parameter into unified memory or VRAM, InferSSD reads the experts it needs from disk. The practical result is that a model far larger than available memory can still answer, but some speed depends on the SSD and the memory budget. The README positions the tradeoff as capacity over fit, not as instant performance.

Supported models and paths

Currently the project documents two tested GGUF checkpoints: DeepSeek-V4.1-Flash-1017 (763B total, 8–16B active) and Muse-Spark-Open (816B total, 42B active). The DeepSeek Q8_0 conversion keeps routed experts at native 4-bit MXFP4 and stores FP8 tensors as 8-bit Q8_0; format details are in the benchmark documentation. Other models are described as forthcoming, so this is not a general-purpose GGUF runner yet.

Once installed, inferssd cli provides a terminal chat, while inferssd serve starts a web UI and an OpenAI-compatible API at the local address, so existing OpenAI-compatible clients can point at /v1. The first run downloads roughly 470 GiB of model files unless you point it at an SSD with --model-dir or supply existing shard files with -m. There is no memory-size setting; the engine uses what is available.

Requirements

Apple Silicon macOS and Linux with an NVIDIA GPU are the install targets; Windows is marked in progress. The one-command installer builds from source because prebuilt packages are not yet available, taking about 2 to 12 minutes depending on the machine. It installs privately without sudo and needs Git, Python 3.9+, a C/C++ compiler, and on macOS the Xcode Command Line Tools. Linux additionally needs the CUDA toolkit, liburing headers, and systemd with cgroup v2. Upgrade and uninstall instructions are kept in the quick-start guide.

Performance scope

The published benchmarks cover 4K prompts and 256 output tokens, with decode measured over tokens 129–256, on four specific machines. They are token-per-second rates compared against llama.cpp and, on the MacBook Pro, Colibri. These are engine-level measurements under fixed conditions, not end-to-end application throughput on arbitrary hardware. The README presents large speedups, but the evaluation is limited to the two named models and those four configurations.

Licensing

InferSSD is Apache-2.0; bundled llama.cpp/ggml and other third-party components retain their own licenses and notices. Model weights remain under their publishers' licenses, so check the Hugging Face model pages before redistribution or commercial use.

Project source and documentation