Find your next open-source project

Search by name, purpose, or keyword — or paste a GitHub repository URL

vllm-project/vllm-ascend

Machine Learning

Hardware plugin that runs vLLM inference on Huawei Ascend NPUs for serving LLMs and multimodal models.

3,107
2,761
Python
Apache-2.0
1 year old

What it is

vLLM Ascend is the community-maintained hardware plugin that lets vLLM serve models on Huawei Ascend NPUs. It follows vLLM's hardware-pluggable RFC, so the Ascend backend is decoupled from vLLM core, and supports Transformer-style LLMs, Mixture-of-Experts models, embedding models, and multi-modal LLMs Overview. It is aimed at teams deploying LLM inference on Ascend hardware rather than on GPUs.

Prerequisites

The supported hardware list covers Atlas 800I A2/A3 series, Atlas 300I Duo, and 950PR/950DT products, on Linux. Software must be pinned to a compatible set: Python 3.12, CANN 9.2.1 (matching your Ascend HDK release), PyTorch 2.10.0 with TorchNPU 2.10.0.post7, and a vLLM version identical to the vllm-ascend version you install Prerequisites. Model and feature coverage is listed in the Support Matrix.

Quick start

  1. Confirm your machine matches the hardware and software prerequisites above, and check the Ascend HDK release notes for the CANN version.
  2. Choose a release. The maintainers recommend the latest stable version v0.30.0; v0.23.0 is also listed as stable Getting Started.
  3. Follow the official installation instructions for that version, which install the plugin against the matching vLLM build: Installation.
  4. Then follow the version-specific QuickStart to launch vLLM with the Ascend plugin and verify that a model serves requests.

If you do not have Ascend NPU resources, the README points to HiDevLab's online development service for applying for access HiDevLab. The exact pip/container commands and the resulting server output are version-specific and live in the linked installation and QuickStart pages, not in the README.

Versioning and branches

The main branch tracks vLLM main and is CI-tested against vLLM main and the v0.33.0 tag. Each vLLM release has a matching releases/vX.Y.Z development branch; v0.13.0/0.18.0/0.23.0/0.30.0 are still maintained, with v0.13.0 accepting bug fixes only Branch. Match the plugin release to the vLLM version you run, and consult the versioning policy before upgrading either side. Feature work happens on rfc/<feature-name> branches.

Using it in practice

This is a server-side inference stack: you need Ascend hardware and the full CANN/PyTorch/vLLM toolchain, and the plugin version must match vLLM, so upgrade the two together. Development, testing, and help go through the contributing guide, the issue tracker, and the user forum; weekly meetings are held Wednesdays 15:00–16:00 UTC+8 Weekly Meeting. The project is Apache-2.0 licensed LICENSE.

HotRepo's take

The project appeared in HotRepo's Python daily snapshot on 2026-11-18 with a best position of 13 and 13 stars gained that day, alongside 3,105 stars and 2,758 forks. The near-parity of forks to stars is notable for a hardware plugin: it suggests people are cloning and adapting it to their own Ascend deployments rather than only starring it. The single recorded daily snapshot is not enough to describe a growth trend, but its ranking shows it drawing attention among Python repositories that day.

Written by AI from the project's README, updated . The project's own docs are the source of truth.

You might also like

  1. vllm-project/vllmServes and batches LLM inference at high throughput for teams running their own models.Python95,320
  2. kserve/kserveServe LLM and classical model inference on Kubernetes through one standardized, autoscaling platform.Go6,173
  3. sgl-project/sglangServing stack for teams that need fast LLM, VLM and diffusion inference at scale on many hardware backends.Python37,754
  4. deepseek-ai/DeepInferServes DeepSeek's own inference stack to teams running DeepSeek-V4.1-Flash at production scale on NVIDIA or Ascend hardware.Python21,703
  5. deepseek-ai/FlashMLAProduction attention kernels for DeepSeek's MLA and sparse attention, aimed at teams serving DeepSeek models on modern NVIDIA or Ascend hardware.C++13,180