FutureMLS-Lab/FlashStates

FlashStates

FlashStates is a research serving stack for agentic LLM inference built on a vLLM v0.33.0 fork and an optional Mooncake store fork. It targets hybrid models that mix attention with linear-attention or state-space layers, where a cached prefix is only reusable together with its hybrid state: the attention KV of every token plus a recurrent checkpoint at the prefix's exact end. Stock vLLM stores recurrent checkpoints only at KV-cache block boundaries, so each agent turn recomputes the end of the previous turn and sometimes whole contexts under load.

The project adds three things: arbitrary state checkpointing at any token position, cost-aware admission and eviction that keeps a recurrent checkpoint and its KV blocks together, and an optional Mooncake host-memory tier that writes through each finished request's hybrid state. It is intended for teams serving coding agents or other growing-context workloads on hybrid models, not for general-purpose or non-hybrid serving.

Quick start

You need NVIDIA B200 or GB300 GPUs, Docker with the NVIDIA container runtime, and model weights on the host. Pull the image matching your hardware:

GPU	Models	Image
B200	GLM-5.3-Flash, Qwen3.8-Flash-Next, Nemotron 3 Super	ghcr.io/futuremls-lab/flashstates-b200:v0.1.0-b200
GB300	Kimi-K3.1-Pro	ghcr.io/futuremls-lab/flashstates-gb300:v0.1.0-kimi-k31-gb300

Serve GLM-5.3-Flash on two B200 GPUs, with /models/glm53flash holding the weights:

docker run --rm --gpus '"device=0,1"' --ipc=host --network host \
    -v /models/glm53flash:/model:ro \
    ghcr.io/futuremls-lab/flashstates-b200:v0.1.0-b200 \
    vllm serve /model --tensor-parallel-size 2 --enable-prefix-caching \
    --mamba-frontier-checkpoint --mamba-prefill-export \
    --prefix-cache-eviction gds --prefill-cost-coefficients 1.862e-5 3.33e-6 \
    --prefix-cache-leaf-last --prefix-cache-leaf-idle-seconds 60

Clients then use vLLM's OpenAI-compatible API unchanged. The prefill cost coefficients are model-specific; per-model settings are in the benchmark configs, and each FlashStates flag is explained in the changes document. Kimi-K3.1-Pro instead runs one engine across two GB300 nodes, documented on its branch.

To reproduce a benchmark point, the scripts start a fresh server and replay a fixed number of agent sessions:

MODEL_DIR=/models/glm53flash HF_HOME=/data/hf GPUS=0,1 PORT=8000 \
    benchmarks/run_point.sh glm53flash-nvfp4-gpu flashstate 48 results/glm53flash-nvfp4-gpu/flashstate/48

Each point takes about 50 minutes on GLM-5.3-Flash and about 80 minutes on the other models.

Constraints

• Hardware is restricted to B200 for the three-model image and GB300 for Kimi-K3.1-Pro; the documented results use two B200 GPUs per server or eight GB300 GPUs for one Kimi engine. Other GPUs need manual KV-cache sizing.
• The flags and options are research code and may change between releases.
• The prefill cost coefficients must match the model; using the GLM-5.3-Flash values for another model would make cache admission decisions incorrect.
• Building from source requires compiling both the vLLM fork and the Mooncake fork; the project also derives its replay client from AIPerf and agentx-harness, with third-party notices listed.
• License is Apache 2.0.

HotRepo's take

FlashStates entered HotRepo's new-project list on 2026-11-10 with 100 stars and 3 forks gained in roughly 23 hours from a 2026-11-09 creation, placing it on the Python new list. One snapshot is not enough to judge sustained growth, but the reported numbers are concrete: up to 3.02× vLLM decode and 3.14× prefill throughput, 96% fewer median uncached prompt tokens and 88% lower p90 time to first token on GLM-5.3-Flash at 48 concurrent sessions. Those claims depend on two B200 GPUs and a specific replay setup, so they describe the benchmark conditions rather than general speedups.

Project source and documentation