StargazerX0/SpecPilot

What it is

SpecPilot targets a practical gap in speculative decoding: released drafter systems leave speed on the table, and the best fix differs by deployment. The project frames this as an agentic optimization loop — a cost model finds the largest gaps between the current system and its ceiling, parallel agents explore and integrate promising directions, a kernel sub-agent tunes kernels furthest from memory-bandwidth limits, and token forcing checks every emitted token against the stock system.

What the repository actually ships is the discovered engines for six deployments, plus scripts to serve, benchmark and verify them. The six settings cover Qwen3.8-27B and Muse-Glimmer-30B with DFlash2 or DSpark drafters on H100 80GB and RTX PRO 6000 Blackwell, including three batching settings with FP8 Qwen. The README claims every "final" engine is lossless: decoding is greedy and the target model decides each token. The main single-request mechanism is W12, a 12-bit weight format that decodes bit-exactly inside the GEMM and is reported to make plain BF16 decoding 1.15–1.26× faster at batch 1–32.

Quick start

The documented flow assumes one of the listed GPUs (H100 sm_90 or RTX PRO 6000 sm_120), Python 3.11 with torch 2.13 + CUDA 13.0, and CUDA/Triton kernel compilation. From a fresh clone:

# 1. Environment (once per machine)
bash scripts/setup_env.sh

# 2. Download checkpoints for one setting
bash scripts/download_models.sh h100_c1_qwen38-27b_dflash2

# 3. Serve the SpecPilot final on GPU 0, port 30000
bash scripts/serve.sh h100_c1_qwen38-27b_dflash2 final 0

# 4. Benchmark, then stop
bash scripts/bench.sh h100_c1_qwen38-27b_dflash2 final-1 0
bash scripts/stop.sh 30000

Repeat steps 3–4 for stock and ar, then compare from the bench directory:

cd ~/specpilot_data/work/bench
python3 ~/SpecPilot/scripts/compare.py speed --ar ar-1.json --stock stock-1.json --final final-1.json
python3 ~/SpecPilot/scripts/score.py final-1.json stock-1.json

Setting names are folder names under settings/. Data defaults to ~/specpilot_data (override with SPECPILOT_DATA); shared/ holds the venv, checkpoints and caches, work/ holds assembled engines, logs and results. Servers expose SGLang HTTP and OpenAI-compatible APIs on 127.0.0.1:<port>. The first boot compiles CUDA, Triton and DeepGEMM kernels; the README warns that a first measured pass on a fresh install ran 10–15% slow on batching at 16 requests, so warm up once before measuring.

Results and verification

For single-request H100 settings, reported SpecPilot/stock speedups are 2.24× (Qwen + DFlash2) and 2.52× (Qwen + DSpark); the four H100 settings were reproduced from a fresh clone with 95% bootstrap confidence intervals, e.g. 2.25× [2.18, 2.32] and 2.52× [2.46, 2.58]. Batching settings yield roughly 1.9–2.0× over stock. Benchmark scores are explicitly a sanity check, not an accuracy evaluation: many batching answers are cut at 2048 tokens, and stock itself varies by 2 between runs.

scripts/bench.sh runs first turns of the dev splits (gsm8k, humaneval, math500, mbpp, mt-bench), greedy, 2048 new tokens; scripts/compare.py speed reports speedups and output identity; scripts/score.py grades with \boxed{} extraction and sandboxed code execution. Outputs are not bitwise identical to autoregressive decoding for any speculative decoder, so acceptance relies on a token-level check that each committed token is the target's own choice.

Constraints

• Each setting runs on one GPU of the listed type. RTX PRO 6000 kernels are built for sm_120a only; H100 finals were measured on H100 only.
• torch 2.13 + CUDA 13.0 needs an R580 driver; older drivers use NVIDIA forward-compat libcuda via setup_env.sh.
• All checkpoints are public on Hugging Face at pinned revisions; download_models.sh fetches exactly those.
• The Muse chat template embeds the current date in the system prompt, so all arms of a comparison must run within one UTC day.
• The build is stock SGLang pruned to the used paths (python/) with setting-specific files copied on top; scripts/build_engine.py assembles and hashes a tree.
• Apache-2.0; python/ is derived from SGLang.

HotRepo's take

The repository collected 48 GitHub stars in the roughly 18-hour window tracked on the Python new-projects list, reaching 151 stars and 9 forks by 2026-11-05 13:45 UTC, with no all-language daily rank recorded. That is a fast but short observation window, and the value proposition here is unusually concrete for that kind of growth: six named deployments, pinned checkpoints, per-setting mechanism lists, reproduction scripts and confidence intervals rather than a single headline number. The combination of W12, token-level losslessness checks and a reproducible bench harness gives readers something to measure locally; the main caveat is that the setup is strictly tied to H100 80GB and RTX PRO 6000 hardware, so most of the claimed 2× gains cannot be evaluated elsewhere.