[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"$f20nhl0kb0gz6f":3},{"code":4,"message":5,"data":6,"timestamp":32},10000,"success",{"full_name":7,"position":8,"description":9,"language":10,"stars":11,"forks":12,"stars_gained":13,"age_days":14,"hn_threads":15,"one_liner":22,"summary_md":23,"topic":24,"github_topics":27,"license":28,"homepage":29,"updated_at":30,"history":31},"deepseek-ai\u002FDeepInfer",1,"DeepInfer: High-performance, large-scale inference framework for DeepSeek models","Python",21741,2469,2571,8,[16],{"id":17,"title":18,"points":19,"comments":20,"posted_at":21},50163277,"DeepInfer: High-performance inference framework for DeepSeek models",2346,891,"2026-10-20","Serves DeepSeek's own inference stack to teams running DeepSeek-V4.1-Flash at production scale on NVIDIA or Ascend hardware.","## The problem\nRunning a frontier MoE model at scale is not the same problem as running a small open model on one box. You need expert parallelism with load balancing, long-context prefill that does not collapse under a large KV cache, and decoding that stays fast as concurrency climbs. General-purpose servers handle all of these, but generally rather than optimally: the kernels, scheduling, and parallelism layout are tuned for many model families, not one. DeepInfer is DeepSeek's own serving stack for its models, released as offline inference code, with the core engine features needed for large-scale serving.\n\n## What's different\nIt is built around a single model family, DeepSeek-V4.1-Flash, and that narrowness is the point. The parallelism scheme (attention DP with balanced CP, MoE EP plus EPLB), the custom kernels (FlashMLA, DeepGEMM, DeepEP, TileLang kernels), and decoding tricks like DSpark speculative verification are all co-designed for the same architecture. The README credits vLLM as its original base, with inspiration from SGLang and TensorRT-LLM, so much of the machinery will look familiar; the difference is tuning, not architecture. Two capabilities stand out for evaluation work: batch-invariant inference and deterministic sampling, which make outputs reproducible across batch compositions -- something most serving stacks do not guarantee.\n\n## Who it's for\nTeams already committed to serving DeepSeek-V4.1-Flash and willing to run on NVIDIA SM100 or Huawei Ascend 950. It is interesting to infrastructure engineers who want the reference implementation the vendor itself uses, and to researchers who need bit-reproducible decoding. Everyone else should look away: this is not a drop-in replacement for vLLM across your model zoo, it is not a local or single-GPU tool, and the README is explicit that a complete service still needs a KV cache system, prefill\u002Fdecode disaggregation, and cross-replica load balancing that are not in this release.\n\n## Cost of adoption\nSteep. Linux only, GCC 13+, Python 3.12, CUDA 13 or CANN 9.2, and a build from source via `.\u002Fbuild.sh` with `DEEPINFER_ACCELERATOR` exported. Benchmarks in the README were measured on 4 SM100 GPUs for prefill and 64 for decode, which tells you the intended hardware envelope. The checkpoint downloads from Hugging Face on first run. The release is eight days old; the README is functional but thin on operational detail, and the missing serving components mean real deployment work remains yours. Apache-2.0 is permissive, so licensing is not the constraint -- hardware and engineering time are.\n\n## Verdict\nThe repo has roughly 21.7k stars in eight days, adding about 2,613 per day and hitting #2 on the new-repo board, plus a 2,346-point Hacker News thread. That is a launch spike driven by the DeepSeek name, not a track record; there is no board history before late October 2026 and no prior maintenance signal. The engineering is credible -- it is the vendor's own stack -- but treat it as a reference implementation for a specific model and specific accelerators, not as a general inference server. Worth reading and benchmarking against vLLM if you already run DeepSeek at scale; not worth adopting on a whim.",{"slug":25,"name":26},"machine-learning","Machine Learning",[],"Apache-2.0",null,"2026-10-24T17:05:24Z",[],1792868441]