NVIDIA/TensorRT-LLM

Overview

TensorRT LLM is NVIDIA's open-source library for accelerating inference of large language models and visual generation models on NVIDIA GPUs. It combines hand-tuned kernels for common operations (attention, GEMMs, MoE), runtime optimizations such as prefill-decode disaggregation, wide expert parallelism and speculative decoding, and a high-level Python LLM API for everything from single-GPU to multi-node deployments.

Built on PyTorch, the library is designed to be modular: models are defined in native PyTorch code, so teams can extend or experiment with the runtime. It integrates with NVIDIA Dynamo and Triton Inference Server for serving, and supports popular models with dedicated example paths, including DeepSeek variants.

The target audience is teams deploying LLM or visual generation inference at scale on NVIDIA hardware, rather than end users looking for a chat app.

Getting started

The README does not include an install command; the official entry points are the Quick Start Guide and the Installation Guide. A typical flow is:

1. Check the support matrix for compatible GPUs, CUDA, PyTorch and Python versions before installing.
2. Install the package per the installation guide, then verify the LLM API and model examples.
3. Run a model through the quick-start example or a model-specific example such as DeepSeek's under examples/models/core/deepseek_v3.
4. For serving, use trtllm-serve; benchmarking is documented in the performance guide.

Constraints

• Requires NVIDIA GPUs and a CUDA/PyTorch/Python stack within the supported version ranges; check the support matrix for exact combinations.
• Some optimizations, such as speculative decoding and wide expert parallelism, target recent architectures; availability depends on GPU generation and model.
• Telemetry is enabled by default and collects anonymized usage and configuration data. It can be disabled with environment variables (TRTLLM_NO_USAGE_STATS=1, DO_NOT_TRACK=1, TELEMETRY_DISABLED=true), the file ~/.config/trtllm/do_not_track, TelemetryConfig(disabled=True), or --no-telemetry on trtllm-serve, trtllm-bench and trtllm-eval.
• Deprecated APIs get a 3-month migration window before removal, with warnings during that period.
• The repository uses the Apache 2.0 license per the README badge.

HotRepo's take

On the 2026-11-06 Python daily list the project recorded a rank of 4 with 10 new stars, alongside 15,026 total stars and 2,913 forks. That is a modest daily gain for a project of this size, and with only a single snapshot there is no basis to describe a trend. The more useful signal for readers evaluating it is the March 2025 HN discussion on the runtime going open-source, where one commenter noted that the previously binary-only Executor component handles request scheduling and KV-cache management, comparable to vLLM or SGLang servers. That context helps explain what the open-source release actually changed for self-hosted serving.