magnitudedev/magnitude

Magnitude is an open-source local inference engine for open-weight models. Instead of shipping one general-purpose set of kernels, it tunes low-level computation for each model on the machine where you download it, aiming to give faster local model serving to people who pair open models with coding agents.

What changes in practice

When a model is downloaded, Magnitude spends up to a minute on the actual hardware trying different ways to distribute the math — thread team sizes, chunk sizes, prefetch distance — and keeps the fastest configuration that does not change model output. This happens per model, so moving to a new machine or a new model triggers new tuning. The engine also assigns a speculative-decoding drafter to each catalog model and quantizes the KV cache to 8-bit keys and 4-bit values, which the author says reduces per-agent memory and is released when agents stop README. Sessions share prefix caches so concurrent agent sessions do not slow each other down.

The practical interface is a desktop app plus a CLI; a single button connects supported agents such as Pi, OpenCode, Hermes, Codex, Claude Code, and Cline, and anything else can connect through an OpenAI-compatible API README. On Linux and macOS, magnitude server setup installs it as a boot-time service that keeps running after logout, self-updates, and serves browsers and other agents over the network — this is the path relevant to remote or shared use, documented separately in the remote-server guide remote server docs.

Performance claim and its basis

The headline "2x faster than Ollama" is an author-reported average across 11 popular local models on Mac and NVIDIA: about 1.32–1.34x faster prefill and 2.17–2.19x faster decode, comparing Magnitude with Ollama benchmarks. Those are engine-level throughput figures under the author's benchmark setup, not end-to-end agent task times, and they are not an independent measurement. In the launch discussion, a commenter noted that several optimized local engines already exist and that benchmarks at short context may not reflect behavior at large context; a maintainer replied that speculative decoding, KV-cache memory, and the listed areas are what the project specifically targets HN thread.

Requirements and constraints

macOS, Linux, and Windows are supported. Hardware can be Apple Silicon, NVIDIA, AMD, or Intel GPUs, or CPU only; there is no fixed minimum, but smaller machines run smaller models and more memory allows larger ones README. Model coverage is not universal: optimized kernels exist for popular open-weight families, with the full list at the models page models. Privacy is local by default — prompts, files, and models stay on the machine and no internet is needed once a model is downloaded README — though the server path deliberately exposes it to other computers, so that boundary is the user's choice. The project is Apache-2.0 licensed repository.