Betelgeuse-AI/Betelgeuse

What It Is

Betelgeuse is a native C++/CUDA fork of llama.cpp aimed at running frontier-scale mixture-of-experts models on a single consumer GPU. The stated envelope is an RTX 3060 with 12 GB VRAM, 16 GB of system RAM and a local NVMe drive.

The approach keeps dense layers, attention, shared experts and routing heads on the GPU while routed experts live in a bounded VRAM slot pool. Experts that are not resident are read from NVMe through io_uring, cached in pinned host memory, and promoted into VRAM when needed. The README insists every routed expert is computed at full quantized precision, with no pruning, skipping or requantization (pipeline overview).

Two executables are produced: bel-server, an OpenAI-compatible HTTP server, and bel-cli, an interactive chat client. Upstream C API and CMake target names are kept; the build targets are still llama-server and llama-cli and the output binaries are renamed.

Supported Models

Two configurations are documented: a 748B-parameter DeepSeek-V4.1-Flash (1017) setup where the backbone and a large Engram embedding table stream from disk, and Alpha-Orionis-XS, an approximately 125B-parameter quantization of Qwen3.8 Flash Next packaged as a four-file bundle around 68 GB (model comparison). The Qwen bundle must be fetched from Betelgeuse-AI/Alpha-Orionis-XS; its model card and weight licenses apply separately from the engine's MIT license.

The DeepSeek setup uses a DSpark speculative drafter; the README states no speculative speedup is certified for the Qwen bundle.

Build

The paged runtime targets Linux x86-64, CUDA sm_86 (RTX 3060 12 GB) and local NVMe. You need a C++17 compiler, CMake, the CUDA toolkit and driver, liburing development files, and the usual llama.cpp build dependencies. Building the embedded terminal logo also needs Python 3, rsvg-convert and ImageMagick, though those are not needed at runtime (build notes, build dependencies).

cmake -S . -B build \
    -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 \
    -DLLAMA_BUILD_TOOLS=ON

cmake --build build --config Release --target llama-server llama-cli -j6

The resulting binaries are build/bin/bel-server and build/bin/bel-cli. Upstream llama.cpp binaries and container images do not include the paged runtime.

Quick Start

Download the Alpha-Orionis-XS bundle and place it in a local directory:

hf download Betelgeuse-AI/Alpha-Orionis-XS \
    Qwen3.8-Flash-Next-Dense.gguf experts.bin \
    Qwen3.8-Flash-Next-PLE.gguf profile-qwen4exp.bin \
    --local-dir /path/to/Alpha-Orionis-XS

export BUNDLE=/path/to/Alpha-Orionis-XS
export MOE_HOT_K=0 MOE_PREKICK=0 MOE_SEED_BUDGET_MS=0

Start the server inside a 12 GiB memory cgroup with swap disabled, which matches the recorded test setup:

systemd-run --user --scope --quiet -p MemoryMax=12G -p MemorySwapMax=0 \
    build/bin/bel-server \
    -m "$BUNDLE/Qwen3.8-Flash-Next-Dense.gguf" \
    --moe-direct-io --moe-pack-dir "$BUNDLE" \
    --moe-profile "$BUNDLE/profile-qwen4exp.bin" --moe-slots 5000 \
    -ngl 99 -np 1 -c 512 -ub 64 -t 6 \
    --cache-warmup 0 --host 127.0.0.1 --port 8080

Then send a request to the server:

curl -s http://127.0.0.1:8080/completion \
  -H "Content-Type: application/json" \
  -d '{"prompt": "def binary_search(arr, target):\n", "n_predict": 68, "temperature": 0.0, "ignore_eos": true}'

The first pass pays disk-streaming cost; warm passes are the throughput figures the README reports (server startup, request example). For interactive use, run build/bin/bel-cli with the same model and paging flags, then type at the You > prompt; /clear resets the conversation and /exit quits (CLI usage).

Constraints

• One paged context per process: keep -np 1. Parallel contexts are not safe with the shared VRAM slot pool.
• Custom CUDA kernels target sm_86 only. Other GPUs, CPU-only builds, non-Linux hosts and other backends are not certified for paging.
• Direct I/O needs contiguous, uncompressed sector access. On Btrfs, the README instructs running chattr +C on model directories before copying weights.
• The recorded Qwen configuration is -c 512 -ub 64 -t 6 -ngl 99; larger contexts or batches need their own memory and correctness checks.
• Cold starts stream a new expert footprint from disk; peak throughput only appears on warm passes with a populated resident set.
• The host victim cache defaults to 8 GiB and counts toward process memory; resize or disable it via MOE_CACHE_MIB.
• Model weights are not covered by the MIT engine license. The Qwen source license includes commercial and AI Work Assistant restrictions; DeepSeek weights and the drafter have their own terms (licenses, Qwen license).

HotRepo's take

The repository recorded 111 stars at its single new snapshot on 2026-11-07, gaining 9 stars in a one-hour growth window, and has not appeared on the all-language daily rank. That is an early signal only: one snapshot on a short window cannot establish a trend.

The substance to watch is whether the paged runtime and the claimed warm throughput (18.92 tok/s on the DeepSeek setup, 22.03 tok/s on the Qwen bundle) survive independent reproduction, since the README itself stresses these are separate historical runs rather than paired benchmarks (measurement caveats). The project is also tied to a single GPU class and a single storage layout, so its practical value for most readers depends on how far that certified envelope widens.