JakeATX/llamAmpere

Project overview

llamAmpere is a fork of llama.cpp that targets NVIDIA RTX 30-series (Ampere) GPUs, particularly the RTX 3090 and 3090 Ti, for local inference of Qwen3.8 models. It provides its own quantization and weight formats, speculative decoding through an MTP draft head, Flash-Next expert offloading between GPU and system RAM, and quantized KV cache options. The project is aimed at practitioners who already run llama.cpp-style servers and want concrete speed or context-length gains on Ampere hardware, including configurations that trade system RAM for 200K+ token context on smaller cards.

The repository is MIT licensed and includes third-party libraries inherited from upstream llama.cpp. It serves an OpenAI-compatible API through llama-server, so existing clients can connect without a custom protocol.

Notable capabilities

• Flash-Next support with experts split between the card and RAM, and --kv-gpu-tokens N to keep only part of the context on the GPU.
• AI+ model files, a project-specific weight format for Ampere and newer cards; older AI+ files are no longer compatible after v0.6 v0.6 release notes.
• Speculative decoding with the MTP draft head, plus an adaptive drafting mode in the 200K-context example.
• KV cache quantization and other memory controls for small cards KV cache documentation.
• Windows improvements, including Flash-Next card-side expert tables on native MSVC + CUDA builds, measured by a contributor on an RTX 3090 with 64 GB RAM v0.6.3.9 release notes.

Quick start

One documented Linux path builds llama-server with CUDA arch 86 and runs the Flash-Next model. It needs the CUDA 12+ toolkit (nvcc), CMake, git, a C++ compiler, the hf CLI from huggingface_hub, and about 81.63 GB for the model and drafter plus build and prompt scratch space Quick start. The README says this was tested with a 3090 limited to 12 GB and roughly 90 GiB of system RAM; native 12/16 GB cards remain unmeasured.

The single paste below checks out tag v0.6.3.9, builds only the llama-server target, downloads the model and drafter from Hugging Face, and launches the server through the project's Flash-Next launcher script:

if [ -d llamAmpere ]; then
  git -C llamAmpere fetch --tags --force && git -C llamAmpere checkout v0.6.3.9
else
  git clone -b v0.6.3.9 https://github.com/JakeATX/llamAmpere.git
fi && cd llamAmpere && \
cmake -S . -B build-sm86 -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DLLAMA_BUILD_BORINGSSL=ON && \
cmake --build build-sm86 -j8 --target llama-server && \
hf download jakeatx/qwen3.8-flash-next-Swift1.5-ATX-AI3 --include '*.gguf' --local-dir models/flash-next && \
hf download jakeatx/Swift-1.5-Qwen3.8-Flash-Next-MTP-Drafter-GGUF mtp-Swift-Qwen3.8-Flash-Next-GSQ-RCO-Q2S-B.gguf --local-dir models/flash-next && \
LLAMA_SERVER_BIN=build-sm86/bin/llama-server \
FLASH_NEXT_DRAFTER=models/flash-next/mtp-Swift-Qwen3.8-Flash-Next-GSQ-RCO-Q2S-B.gguf \
scripts/flash-next-server.sh -m models/flash-next/qwen3.8-flash-next-Swift1.5-ATX-AI3-00001-of-00002.gguf

After the build, the server listens at http://127.0.0.1:8080 with an OpenAI-compatible API. To restart later, rerun the last three lines from the llamAmpere directory.

A second documented path runs Swift 1.5 27B at 2.3 bpw with 200K context on one 24 GB RTX 30-series card, downloading about 9.1 GB of model data on first start. The README also points to commands for 12 and 10 GB cards that have not yet been run on this model v0.6.2 release notes.

Constraints and caveats

• The primary tested target is one RTX 30-series card on Linux; Windows support is improving but several speed figures are contributor measurements, and the maintainer states they have no Windows machine and built under MinGW/Wine, which cannot compile the CUDA and MSVC parts v0.6.3.9 release notes.
• Small-card configurations rely on system RAM and explicit memory settings; the README does not promise that 64 GB of system RAM is enough for every model/context combination.
• AI+ weight files are specific to this engine and version; older AI+ files no longer load after v0.6.
• Performance numbers in the README are observations from specific cards, prompts, and seeds, not same-engine apples-to-apples comparisons in all cases.
• The project forks upstream llama.cpp, so build flags and server behavior follow that codebase while the Ampere-specific kernels and formats differ.

HotRepo's take

On the all-language daily GitHub leaderboard for 2026-11-09, the project recorded 301 stars and 37 forks with 6 stars gained that day and a best position of 6 on the C++ list. That points to a niche with real attention from C++/CUDA users rather than a broad breakout. The value here is concrete: long-context Qwen inference on a 3090-class card, with published release notes documenting memory behavior and measured token rates. Readers without an Ampere GPU or without a clear need for 200K+ context will find standard llama.cpp simpler, while those chasing throughput on RTX 30-series hardware have a documented starting point.