vixhal-baraiya/qwenGPU

What it is

qgpu augments Qwen3.5-2B with a small SIMT GPU compiled directly into the model's weights. The GPU is 45 additional transformer layers placed before Qwen's own 24 layers; their weights are computed by a compiler rather than learned. The machine has four lanes per warp, 15 registers per lane, a four-stage pipeline, 32 program blocks, and 126 words of global memory. Its state lives in the recurrent state of Qwen3.5's Gated DeltaNet layers, and a launch is represented as a stream of model-generated tokens (overview, GPU state design).

The intended audience is people interested in mechanistic interpretability, compiled computation inside language models, and unusual inference experiments. It is not a general-purpose GPU emulator or a replacement for CUDA hardware.

How a launch works

The host compiles CUDA source to PTX with clang, lowers the PTX into the GPU's machine blocks, and feeds the program image and input arrays to the model as tokens. Qwen then generates the execution records: which block each warp runs, register values, branches, and memory writes. The host reads result arrays from the model's recurrent state but does not compute them. Two modes are available: normal mode expresses arithmetic as operations run by ready-made kernels, while advanced mode has Qwen write the CUDA kernel itself, guided by the qgpu CUDA reference. The terminal chat includes a live panel showing the scheduler, program ROM, pipeline, register file, and memory (how it works, host/weights split).

Quickstart

You need Python 3.11+, uv, just, around 16 GB of free memory, and a machine with Apple silicon, an NVIDIA GPU, or a CPU. For advanced mode, install a clang with the NVPTX backend, such as brew install llvm, or point the CLANG environment variable at it.

git clone https://github.com/vixhal-baraiya/qwenGPU && cd qwenGPU
just setup
just build
just chat

just build downloads Qwen3.5-2B and compiles the GPU into it. In the chat, enter / to see commands. Other entry points include a one-shot bridge, a merged-model comparison check, and a chat evaluation script (quickstart, alternative commands):

uv run python -m qgpu.bridge build/merged-2b "What is the sum of 12, 7, 30 and 5?"
uv run python scripts/merged_check.py models/Qwen3.5-2B build/merged-2b
uv run python scripts/eval_chat.py build/merged-2b --samples 3

Results and limits

On Apple silicon with MPS and bf16, the author reports exact token-level matches between the model's launches and a reference interpreter, and that interpreter matched a T4 on 49 test kernels. Text behavior is said to stay within a KL divergence of about 7e-4 of the original model. Normal mode answered 20 of 24 sampled chat questions with GPU-computed numbers, but advanced mode produced correct kernels for only 4 of 5 simple vector and matrix questions on the first try. The README explicitly calls a 2B model's CUDA unreliable and recommends normal mode for answers you depend on (results).

A launch takes tens to a few hundred generated tokens, at roughly 10–15 tokens per second on MPS. The project requires the merged checkpoint built by just build; it is not a drop-in inference server for the original Qwen model. The code is MIT-licensed, while a merged checkpoint inherits Qwen3.5's Apache-2.0 terms because its config is widened and extra layers are added (license and checkpoint terms).

HotRepo's take

qgpu had a modest new-project record on HotRepo: 108 stars and 16 forks, with 7 stars gained in the tracked window on 2026-11-04. Those numbers describe early attention rather than adoption. What makes the repository worth reading is the concrete mechanism behind the demo: a compiler-produced GPU circuit, PTX lowering, a token-encoded execution log, and a replay check against a reference interpreter. The author's own caveats about queued CUDA correctness are the important boundary for anyone evaluating it as a calculator.