FlashML-org/FreeToken

What it does

FreeToken is an edge-native serving engine for Mixture-of-Experts models. Instead of requiring a datacenter node, it treats the GPUs, CPU, host memory, and interconnects of a personal machine as one elastic inference platform, so open-weight models in the hundreds of billions of parameters can run on a gaming desktop or laptop.

The engine combines bandwidth-adaptive CPU–GPU co-execution, double-buffered prefill streaming, global LRU expert caching, and the FTW weight format. It also keeps semantic anchor checkpoints for recurrent state and KV cache, so agentic context edits such as tool calls and thinking blocks do not force a full context recompute. VRAM can be re-split at runtime between expert caches and KV memory without restarting the engine or reloading weights.

It is aimed at people who want to run large open-weight MoE models locally and connect them to coding or tool-calling agents. Supported model families listed in the README include DeepSeek-V4-Flash, Qwen3.6-35B-A3B, and GLM-5.2, in MXFP4, NVFP4, FP8, and BF16 formats, and the server exposes Anthropic/OpenAI-compatible APIs. The design follows work from SGLang, vLLM, FlashInfer, and llama.cpp; the method is described in the FreeToken paper.

Install and run

Two entry points exist. The desktop app for Windows and Linux is downloaded from flashml.ai; it configures the engine and provides a GUI for running models, chatting, and tuning settings. The CLI path uses uv (recommended) or pip:

uv pip install "freetoken[accel]"

To build from source:

git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken
uv venv && source .venv/bin/activate
uv pip install -e ".[accel]"

After installation, use the quick start guide to launch a model and the CLI reference for command options; the README does not inline the serving command, so start from those documents. Model selection and download details are in the supported models list, and platform-specific prerequisites are in the install guide.

Requirements and limits

• The runtime targets consumer hardware: laptops, gaming desktops, and workstation GPUs, with native support for NVIDIA RTX 30, 40, and 50 series cards. Other GPU vendors are not listed as supported.
• Only the desktop app names Windows and Linux as platforms; macOS is not mentioned.
• Large models need enough combined VRAM, system RAM, and disk for their weights and caches; the engine pools these resources but does not remove the capacity requirement.
• Loading failures and older FTW checkpoints have dedicated documents: checkpoint diagnosis and FTW hotfix.
• The project is Apache-2.0 licensed.

HotRepo's take

FreeToken entered GitHub's all-language daily chart on 2026-11-01 at position 16 with 232 stars gained that day, and the repository now sits at roughly 15,000 stars and 1,500 forks. A Hacker News thread from 2026-08-21 titled "Run 290B+ frontier MoE models locally on your gaming PC" drew 41 points and 3 comments; one commenter said the repo and site left them unsure what the project actually does (guax), while another pointed readers to the paper as the more informative artifact and found the GitHub presentation unclear about the targeted optimizations (gravypod). Those remarks line up with the README's structure: the technical substance is in the paper and the linked docs, while the repository landing page emphasizes headline capability. Readers evaluating it should read the paper and the quick start rather than the one-line pitch.