Helldez/BigMoeOnEdge

What it does

BigMoeOnEdge runs Mixture-of-Experts models in GGUF format that are larger than the machine's RAM. Only a few experts are active per token, so the engine keeps the always-used weights and the hottest experts in memory and reads the rest from flash on demand. Output is claimed to be identical to running the whole model in memory, and the stack is built on llama.cpp's public API, so existing quantizations, tokenizers and chat templates work unchanged. Typical targets are phones, laptops and consumer GPUs, but the same code also runs on data-center cards. It ships as a desktop app, Android/iOS apps and CLI/server binaries; an --auto planner measures the hardware and picks cache sizes, threads and device placement (How it works).

Quick start

Pick a MoE GGUF model and install the command-line tools from the latest release with the install script:

curl -fsSL https://github.com/Helldez/BigMoeOnEdge/releases/latest/download/install.sh | sh

On Windows use the PowerShell variant instead:

irm https://github.com/Helldez/BigMoeOnEdge/releases/latest/download/install.ps1 | iex

Then let the planner choose settings and run a first prompt:

bmoe-cli -m model.gguf --auto -p "Explain MoE routing."

bmoe-cli -m model.gguf --plan prints the plan without generating. To expose the model to other tools, start the server, which also serves a web UI on port 8765; Android, iOS, macOS, Windows and Linux apps are available from the latest release:

bmoe-server -m model.gguf -c 32768

The server speaks the OpenAI and Anthropic APIs, so an OpenAI SDK client can point at http://127.0.0.1:8765/v1. Context sizing and agent integrations are covered in the server API docs.

Constraints

• The engine only streams MoE models; which tensors are experts is read from the GGUF file, and non-MoE models or models without identifiable experts may not benefit (which models stream).
• Throughput depends heavily on storage speed and memory. The published table reports roughly 5-6 tok/s for Qwen3.6-35B-A3B on a 12 GB phone and 18-24 tok/s on a 16 GB Apple-silicon laptop; these are single-machine measurements, not guarantees.
• Setup effort is not uniform: macOS builds are not yet notarised, iOS requires building from Xcode and is labelled experimental, and Linux needs WebKitGTK 4.1 for the window (Platforms).
• Coding agents need a large context; 32768 is cited as enough for opencode and 65536 for Claude Code (context sizing).
• License is Apache-2.0, with bundled llama.cpp and other third-party components under their own terms (THIRD_PARTY_LICENSES).

HotRepo's take

On the 2026-11-14 C++ daily list the project recorded rank 7 with 17 stars added, putting a memory-streaming llama.cpp front-end ahead of most C++ entries for that day. The README's distinguishing claim is breadth rather than a single trick: the same streaming engine is presented across Android, iOS, macOS, Windows and Linux, with CPU, Metal, CUDA, HIP, Vulkan, OpenCL and Hexagon paths, and a planner that configures itself. That scope is what makes the project interesting to follow, but the speed figures are first-party measurements at specific memory and storage configurations, so the headline tok/s numbers should be read as per-machine observations rather than expected performance on arbitrary hardware.