llm-d/llm-d-router

llm-d Router is the intelligent entry point for inference traffic in a Kubernetes-based LLM serving stack. It combines a proxy (typically Envoy) with an Endpoint Picker (EPP) that decides which model-server replica should handle each request, based on KV-cache locality, current load and request priority. It fits teams already running inference on Kubernetes who need smarter placement than round-robin.

What it changes in a workflow

Instead of sending every request to the next available replica, the EPP evaluates each incoming request against the current state of an InferencePool and injects real-time signals into the data plane through the ext-proc protocol. Two request-management APIs tune that behavior: InferenceObjective sets scheduling goals such as priority and performance targets, and InferenceModelRewrite rewrites model names to support A/B testing and canary rollouts. A separate Disaggregation Sidecar coordinates multi-stage lifecycles — Prefill/Decode and Encode/Prefill/Decode — by handling KV-cache and embedding transfers between workers (README).

Two deployment paths

Standalone Mode runs a self-managed Envoy proxy without Gateway API infrastructure. In the default sidecar topology the proxy lives in the EPP pod, which the README describes as suitable for testing and local evaluations; service mode runs the proxy as a separate, independently scalable Deployment reachable through the EPP Service. The standalone Helm chart covers both (README).

Gateway Mode is the recommended production path. The EPP becomes a backend for an InferencePool referenced by an HTTPRoute on a shared Gateway, which enables advanced traffic management, multi-cluster balancing and shared infrastructure across inference and traditional workloads (architecture docs).

Requirements and constraints

• You need Kubernetes and, for Gateway Mode, the Gateway API; the InferencePool API lives in the separate Gateway API Inference Extension project, which also defines the Endpoint Picker Protocol (README).
• If you install or configure Envoy yourself rather than using the provided automation, the only supported request_body_mode and response_body_mode is FULL_DUPLEX_STREAMED (README).
• The EPP container sizing guide and TLS documentation are separate documents; heavy or long-context workloads should consult the sizing guidance before provisioning (README).

The README does not publish benchmark results or describe failure, upgrade or rollback behavior, so those must be validated in your own environment. The software is Apache-2.0 licensed, and published container images carry signed provenance attestations and an SBOM with instructions to verify them (README).