FareedKhan-dev/kimi-k3-in-c

What it is

This repository is a from-scratch C99 inference engine for Kimi K3, a 2.78-trillion-parameter mixture-of-experts model shipped as a 1.56 TB checkpoint. The engine has no BLAS, no PyTorch or ONNX runtime, no GPU path, and no third-party numeric library; libm and OpenMP are the only dependencies. It targets people who want to run a very large open model on ordinary hardware, and who are willing to trade speed and storage for the ability to avoid GPUs entirely.

The mechanism behind the small memory footprint is a chain of reductions. Routed experts already ship in MXFP4, so most weights occupy half a byte on disk; since only 16 of 896 experts fire per layer, the expert pool can stay on disk. The always-on dense trunk is packed into a single 109 GB file and streamed through a pinned-prefix-plus-ring layout, which brings measured peak RSS down to 8.24 GB. The README documents the attention variants (KDA and MLA), the router, the expert LRU cache, and the validation gates in unusual depth, including a several-part explanation of why the implementation matches the reference bit for bit.

Quick start

The repository can be built and tested without the checkpoint. This is the recommended first step before committing to the large download:

git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git
cd kimi-k3-in-c
make -j
make test

The test suite needs no weights, no network and no Python, and takes under a minute. It should end with VERDICT: ENGINE MATCHES THE REFERENCE EXACTLY see the quick start.

Running the real model requires the full setup: check the machine with ./scripts/k3-doctor.sh, fetch the checkpoint with a Hugging Face token via ./scripts/download-model.sh ~/k3model, pack the trunk with ./scripts/pack-trunk.sh ~/k3model ~/k3trunk, then generate text:

./bin/k3 ~/k3model --trunk ~/k3trunk --preset workstation \
         --tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental

The tokenizer ships with the checkpoint, which is why --tok points at the model directory full setup steps. Intermediate results: the checkpoint takes about 1.56 TB, the packed trunk about 109 GB, and the first run loads every pinned layer, so it is much slower than steady-state generation.

Usage constraints

• Storage is the hard gate. Plan for roughly 1.7 TB of free space (1.56 TB checkpoint plus 109 GB packed trunk) on fast local disk. The README treats network volumes as a bad fit for throughput and recommends measuring the disk with tools/devbw.py.
• CPU and OS. The reference target is Linux x86-64 with AVX2 and FMA. macOS/arm64 and Windows/x86-64 also build and pass the gates, and make portable builds generic AVX2 without -march=native. AVX-512 is not required.
• Memory presets. --preset selects a trunk/cache split from ultra (~3 GB planned) through laptop (8.2 GB measured peak) to max (~224 GB). All presets stream the trunk and require --trunk; without it the trunk loads resident at about 113.5 GB.
• Generation defaults. Batch generation is greedy and reproducible across memory budgets; chat mode adds opt-in temperature and top-p sampling. The ceiling is 4096 generated tokens and 32768 prompt tokens. For any generation of length, pass --incremental, or each step re-runs the whole prefix.
• Tokenizer quirks. --prompt and --prompt-file require --tok; for non-ASCII text use --prompt-file, because the shell can re-encode argv. The released checkpoint declares two conflicting end ids, so both should be passed via --stop-id.
• Scope. Version one is text-only XTML chat: no tools, vision, HTTP serving or context compaction. The repository ships no model weights; Kimi K3 remains under Moonshot AI's own license, and the code here is Apache-2.0.

HotRepo's take

The project entered the all-language GitHub daily ranking at 11th place on 2026-11-04 with 10,017 stars and 1,606 forks, gaining 351 stars that day. That is a strong single-day result for a repository whose first real workflow is a multi-hour, 1.56 TB download. The depth of the README is also unusual: one Hacker News reader described it as reading like a complete research paper in a GitHub README comment. That fits the material, which mixes build instructions, tensor-level invariants, gate results, memory-ladder measurements and an explanation of why the trunk is left unquantised. The project is therefore best read as both an engine and a detailed systems write-up; the same file that tells you how to run it also tells you why each design choice was made.