NVIDIA/Model-Optimizer

What it does

NVIDIA Model Optimizer is an open-source Python library for applying model optimization methods to deep learning checkpoints. It accepts Hugging Face, PyTorch, or ONNX models and offers post-training quantization, quantization-aware training/distillation, pruning, distillation, speculative decoding, and sparsity, then exports an optimized checkpoint for deployment in TensorRT-LLM, TensorRT, vLLM, or SGLang (project overview). It also integrates with Megatron-Bridge, Megatron-LM, and Hugging Face Accelerate for optimization workflows that need training. It is aimed at teams deploying LLMs, VLMs, and diffusers models who need smaller checkpoints or higher inference throughput without rebuilding the serving stack.

Install and first run

Install the stable package from PyPI with all optional dependencies:

pip install -U nvidia-modelopt[all]

For the latest features or editable development, clone the repository and install it from source:

git clone git@github.com:NVIDIA/Model-Optimizer.git
cd Model-Optimizer
pip install -e .[dev]

NVIDIA container images such as nvcr.io/nvidia/pytorch:<version>-py3 come with Model Optimizer preinstalled; upgrade the package after pulling one. The actual optimization workflow depends on the chosen technique and model family, so follow the relevant example after installation. For post-training quantization of Hugging Face models, start from the HF PTQ example; the installation guide covers dependency variants and environment setup. A run produces an optimized checkpoint that the downstream inference framework can load, but the repository does not provide a single command that covers every model and deployment target.

Techniques and deployment path

Quantization is the most direct path for reducing model size and speeding inference. Quantization-aware training or distillation can recover accuracy after aggressive low-precision conversion. Pruning removes parameters or structured components, while distillation trains smaller models to mimic larger ones. Speculative decoding trains draft modules to propose additional tokens, and sparsity stores only nonzero weights and their locations. Pre-quantized checkpoints published in the Hugging Face Model Optimizer collection are ready for TensorRT-LLM, vLLM, and SGLang (technique guide).

Constraints

Model Optimizer is pre-1.0. Deprecated APIs get a roughly one-month migration window with runtime warnings, after which they may be removed in a minor release. The package pulls in additional third-party open-source dependencies, so their licenses should be reviewed. Compatibility varies by model type and technique; the support matrices for LLM/VLM quantization, diffusers, ONNX, Windows, QAT, pruning, distillation, and speculative decoding are separate and should be checked before selecting a workflow. Exporting for a specific inference framework may require that framework's own installation and version constraints, which are outside this repository.

HotRepo's take

The repository's recorded daily snapshots show a jump from 3,936 stars on 2026-09-24 to 4,313 the next day, with several more hundred-star daily gains through 2026-09-27. By 2026-11-05 it had 5,531 stars and 800 forks, placing 7th in the Python daily list that day. That pattern is consistent with a burst of attention around a mature optimization toolkit rather than a one-day spike. The substantive content—technique guides, per-model-family examples, and published quantized checkpoints—gives the star growth context: readers have multiple concrete starting points for evaluating and reproducing the workflows.