eagle-hq/EAGLE-4

The problem

Speculative decoding promises faster LLM inference, but the usual draft models are tuned for one batch size and one hardware profile. Turn them on under real traffic, scale concurrency to dozens of requests, and the acceptance rate and the tree schedule collapse, so teams either disable speculation or maintain per-workload configurations. EAGLE-4 ships draft models plus eagle-harness, a runtime that measures the cost of drafting on your setup and picks the speculation depth so the draft tree matches batch size, hardware and workload.

What's different

Most speculation setups fix the number of draft steps and the tree shape up front, and the same configuration is applied whether you serve one request or 64. EAGLE-4's own framing is that this version adapts speculation to measured costs and keeps pace with plain decoding up to high concurrency, which is the opposite trade-off from drafting that only pays off for single requests. The project also claims its draft output is exactly what the target model would produce, so speculative decoding does not change answers. It plugs into SGLang, vLLM and TokenSpeed rather than asking you to replace your serving stack, and there is an AMD MI355X path alongside the NVIDIA one. Beyond that, the README is a release announcement: the repository holds the harness and engine integration, not the paper's method in detail, so the algorithmic claims have to be checked against the paper and the support matrix.

Who it's for

Teams already serving open-weight models such as Qwen3.8-27B through SGLang or vLLM, with NVIDIA B300-class or AMD MI355X hardware, who are willing to run their own calibration and want latency and throughput gains without changing output. It suits infrastructure engineers who can test a fork-pinned engine build. It is not for people on consumer GPUs, CPU or small VRAM setups, or anyone who wants model quality or training changes — this is an inference-speed release only. If you just want a drop-in speedup on an unlisted model and engine version, wait until the support matrix covers it.

Cost of adoption

Heavy. The quick start is tested on Linux, a CUDA 13 driver and an NVIDIA B300; pip installs require Python 3.12 plus the CUDA toolkit and a C++ compiler. Serving the sample model and draft downloads roughly 62 GB. Pip users must pin a forked sglang build from a custom wheel index, so engine upgrades are not free. The bigger cost is licensing: the README states this is source-available, not OSI-approved open source, with a separate commercial license needed for a competing offering and a US $20M aggregate revenue threshold for other commercial use. That is a procurement and legal question, not a technical one, and it disqualifies many companies outright.

Verdict

The project is four days old, with about 967 stars and roughly +217 per day on HotRepo's new board — a sharp launch spike, not an established track record. The Hacker News thread drew 2 points and no comments, so outside interest is thin so far. EAGLE is a real line of research (EAGLE-2 and EAGLE-3 are cited), but this release's measured numbers come from the vendor's own four-GPU setup, and the repository is a thin wrapper around forks. Treat it as promising and worth an evaluation on your own workload, not as production-ready infrastructure; the source-available license will decide more deployments than the speedups will.