NVIDIA-NeMo/Switchyard
AI EngineeringA Rust routing library that lets AI agents and LLM gateways pick a model per request.
What it does
Switchyard is an open-source routing library for LLM applications. Instead of sending every request to one model, it helps an agent choose which configured model should handle each request, mixing efficient models with stronger ones to trade off task success, cost, and latency. It is aimed at teams building gateways, agent harnesses, or multi-model services that already own their model clients and credentials.
The core is the Rust library libsy. A routing algorithm inspects the incoming request or recent tool activity in the agent loop, sometimes by calling a decision model, and returns a selection. Your gateway or harness then makes the actual model call. Routing can change as the agent keeps working.
Routing options and components
Four named choices are exposed: Auto (the default preset equal to efficient-first Stage routing with a 0.5 confidence threshold), Task (an LLM classifier judges whether the efficient model suffices), Execution (Stage router uses tool activity and outcome signals), and Composite (combines both). Further strategies such as escalation, advisor, sub-agent, Plan/Execute, and random routing are listed in the routing overview, and integration support varies by gateway.
Components have different maturity: switchyard-libsy is Beta, switchyard-llm-client and switchyard-runner are Alpha, and the standalone proxy switchyard-server is a demo only. All APIs and configuration are pre-1.0 and can change between releases.
Getting started
Pick one of three paths from the README:
- Through an existing gateway. OpenRouter hosts it: set
modeltonvidia/switchyardin API requests. Self-hosted options include the Envoy-based Agent Router (OpenAI-compatible), an experimental LiteLLM plugin that pins LiteLLM 1.102.0, and a native NeMo Relay plugin requiring Relay >=0.10.0,<1.0.0. - Embed libsy. Follow the Rust quickstart or Python bindings. In Rust,
Algorithm::run_streamreturnsStep::CallModel,Step::CallDecision, andStep::Doneitems; your code sends the model requests, feeds typed responses back tocall.respond(...), and returns or forwards the final answer. You supply the algorithm, model pool, request, and agent state. - Evaluate locally. Use the standalone demo server with your own model pool to compare routing configs against a single-model baseline, then verify that routing is actually active. The experimental desktop and terminal quickstart covers local install, coding-tool connections, sessions, usage, and uninstall; integration guides exist for pi, Oh My Pi, VS Code Copilot Chat, and Claude Code.
Constraints
- The standalone server is for demos and local evaluation, not a supported production deployment; production use means embedding the library or using a gateway.
- Pre-1.0 software: pin the version and expect API, configuration, and routing behavior to change.
- Choosing a cheaper model does not guarantee a cheaper completed task; the tuning loop describes how to evaluate the whole agent, model pool, and routing configuration against your baseline. Troubleshooting documentation lists settings that silently disable routing.
- Gateway limits apply, for example the LiteLLM plugin cannot service intermediate model calls needed by classifier or escalation algorithms. Relay 0.8.x and 0.9.0 can lose upstream error status and details when the plugin is enabled.
License and benchmark references
Apache 2.0, copyright NVIDIA Corporation. Benchmark configurations for Terminal-Bench 2.1 and DeepSWE are published alongside the code, and a soak test describes latency and routing-overhead testing.
HotRepo's take
The project has been in the GitHub daily records since August 2026, when an all-language daily listing showed gains around 370–408 stars per day at 668–1,367 total stars. By the Rust-language daily record on 2026-11-17 it had reached 3,615 stars and 374 forks with a sixth-place Rust daily position and 41 stars gained that day. The early all-language growth is the notable part: a multi-model routing library drew broad interest, while its current Rust-board standing reflects that the audience is now narrower. The README's own emphasis that the standalone server is demo-only and the components sit at Beta or Alpha is the main thing to weigh before adopting it.
Written by AI from the project's README, updated . The project's own docs are the source of truth.
You might also like
- experientiallabs/experientialA self-hostable gateway that routes agent traffic across providers, then turns collected traces into custom routers or models.
- jasonxu114514/opencode2apiA self-hosted gateway that puts OpenAI, Responses and Anthropic APIs in front of OpenCode Zen, with key and proxy pooling.
- tashfeenahmed/freellmapiOne OpenAI-compatible endpoint that pools free tiers from 34 LLM providers with routing and failover, for hobby experiments.
- songquanpeng/one-apiSelf-hosted gateway that unifies many LLM providers behind one OpenAI-compatible API, with keys, quotas and load balancing.
- eliasbiondo/antirionA self-hosted gateway that puts routing, budgets, guardrails and observability in front of calls to multiple LLM providers.