Neroued/ninfer

What it is

NInfer is a native C++/CUDA inference runtime for a narrow set of model checkpoints, tuned for one GPU or two DGX Spark nodes. It ships a CLI for one-off generation and an HTTP server exposing OpenAI- and Anthropic-compatible endpoints, plus offline perplexity scoring. The engine adds CUDA Graph decode, speculative decoding (MTP/DFlash), and exact prefix reuse to keep decode and long prefill fast.

The supported table is short and specific: Qwen3.6/3.8-27B dense, Qwen3.6-35B-A3B MoE, Qwen3.8-Flash-Next and GLM-5.3-Flash. Depending on the checkpoint, it runs on a single RTX 5090 (optionally with CPU expert offload), one Spark, or two Sparks in TP2/EP2 configurations. Vision, tool calls, constrained output and offline scoring are supported, but vision and speculative decoding need the matching components inside the artifact and must be enabled at startup.

Build from source

Build natively on Linux. The documented toolchain is CUDA 13.4 with targets sm_120a for RTX 5090 and sm_121a for DGX Spark, CMake 3.28+, a C++20 compiler, Ninja, pkg-config, libjpeg, FFmpeg development libraries, hwloc >= 2.5, liburing >= 2.5 and libcurl >= 7.85.

git clone https://github.com/Neroued/ninfer.git
cd ninfer
cmake --preset rtx5090-release
cmake --build build/rtx5090-release -j

For a single Spark use the dgx-spark-release preset; for two Sparks build on both nodes with -DNINFER_ENABLE_DISTRIBUTED=ON after installing NCCL. Product binaries land in apps/ninfer, apps/ninfer-serve, apps/ninfer-perplexity and, with distributed support, apps/ninfer-worker. Run them from the build tree and keep lib/ next to apps/ if you move them; there is no installed C++ SDK.

Get weights and run

The engine only loads v3 .ninfer artifacts that bundle weights, configuration, tokenizer and chat template. Official artifacts are published on Hugging Face by the same author, e.g. Qwen3.8-27B NVFP4. Download with the hf CLI and keep every part beside the entry file:

hf download neroued/Qwen3.8-27B-nvfp4-NInfer \
  qwen3_8_27b_nvfp4.ninfer --local-dir models

Single request on RTX 5090:

./build/rtx5090-release/apps/ninfer models/qwen3_8_27b_nvfp4.ninfer \
  --prompt "Explain prefill and decode briefly." \
  --max-context 8192 --max-tokens 1024 --kv-dtype fp8

Answer text goes to stdout; reasoning, startup info and the result report go to stderr. For a persistent service, copy a deployment example, set model.path, and start ninfer-serve --config ninfer.local.toml. The serve guide covers endpoints, tools, structured output and streaming; the sample config enables Vision, FP8 KV and MTP3 with a 240,000-token per-request limit but the real shared KV capacity depends on memory.

Constraints

• Hardware is the hard limit: RTX 5090 (sm_120a) or DGX Spark (sm_121a); Spark runs are AArch64 and two-node runs require NCCL on both machines.
• Flash Next is 101.00 GiB across four files and GLM 178.33 GiB across six; both must sit complete on every node, and Flash Next with CPU offload also needs substantial system RAM.
• max_context caps input plus generated tokens per request, while context_capacity is a separate startup-sized pool shared by KV history and continuation state; the pool is not multiplied by concurrency, so not every active request is guaranteed to reach the context limit.
• One Engine serves one model with up to eight active requests; memory pressure can evict cached prefixes or pause requests for replay.
• Model licenses differ from the Apache-2.0 code license: Flash Next combines the Qwen Community License 1.0 with NVIDIA Open Model License weights, while GLM source weights are MIT.

HotRepo's take

NInfer appeared on GitHub's C++ daily list on 2026-11-07 at position 7 with 3,280 stars and 687 forks, gaining 11 stars that day. The single recorded snapshot is enough to place it that day but not to call a trend, and HotRepo holds no full-language daily rank for it. What stands out technically is the deliberate scope: rather than a general runner, it pairs a handful of checkpoints with fixed hardware and documents build presets, per-model artifact sizes and benchmark methodology. That narrowness is also the trade-off — readers with other GPUs or models get no path here, while RTX 5090 and Spark owners get a concrete, reproducible setup.