vllm-project/semantic-router

vLLM Semantic Router sits between an agent harness and multiple model backends. The harness keeps its agent loop, tools and task state, while the Router decides per call which configured model or combination of models to use, based on policy. It is aimed at teams that already use several local, private or cloud models and want routing decisions governed in one place rather than hard-coded in each client.

What changes in practice

Instead of wiring a harness to one model, you point it at the Router's inference endpoint using a public model ID such as vllm-sr/auto (README, agent harness setup). The policy layer then considers the request against preferences such as output quality, latency and cost, and can also select or combine models rather than only choosing one (README). The same layer is intended to keep calls within approved locations, so private and cloud backends can coexist under one routing rule set.

The project also publishes routing/decision models and related blog material, including the Decision and Vela model families and an open foundation routing model line (Hugging Face, Vela 2.0 announcement). Those are part of the Router's own approach; adopting the Router does not require you to train or host them yourself.

Ways the project is used

• Install and route. The documented install path is a shell installer for the stable channel (install script); pip, uv and agent-driven installation are documented separately (installation guide). Setup and compatibility for harnesses are covered by the agent-harness guide (agent harness setup). The README badges indicate Go 1.25 (README).
• Try before deploying. A public online playground with shared credentials is provided (playground). This is a demonstration environment, not a statement about production deployment.
• Operate an inference layer. Topics and release notes describe itself as an AI gateway with guardrails and Mixture-of-Models routing, with release notes describing stateful production routing and fusion (repository, v0.3 release post, fusion post).

Conditions that matter

Routing only works over backends you configure. The Router chooses among "configured backends across local, private, and cloud compute" (README); it does not supply model capacity on its own. Model selection, location constraints and quality/latency/cost preferences are policy choices you define, so routing outcomes depend on the policy you write rather than on an automatic default.

The README does not describe a fallback behaviour when a chosen backend fails, nor an upgrade or rollback path for the Router itself. Those are not evaluated in the supplied material. The project is Apache-2.0 licensed (repository).

HotRepo's take

The only supplied trending observation is 6,310 stars on 2026-10-29 with +28 that day (board data); this single sample cannot establish a trend. The repository was created on 2025-08-26, and the supplied material includes no independent evaluation of routing quality or production reliability.