NVIDIA-NeMo/Speech

NVIDIA NeMo Speech is a framework for researchers and PyTorch developers who work on speech models: automatic speech recognition (ASR), text-to-speech (TTS), and speech LLMs. It combines reusable training and deployment code with NVIDIA's released checkpoints, so you can start from an existing model rather than a blank slate. Since 2026 the repository focuses on audio, speech, and multimodal LLMs; the final pre-split release covering other modalities is v2.7.3.

What you work with

You get model code for creating and customizing speech systems plus open-weight checkpoints published separately on Hugging Face. Recent examples include a streaming ASR model supporting 40 languages with adjustable latency, a bilingual-style multilingual TTS model, and a unified English ASR model that handles both offline and streaming inference. Release notes also reference a speech-to-speech voice chat model offered under early access rather than as ordinary downloadable code. For a concrete workflow, the practical route is to pick a published checkpoint, use the framework to fine-tune or adapt it to your domain, then deploy it with the same code. The optional accelerated backend (Transformer Engine, FlashAttention, Mamba, and similar kernels) needs extra compiled packages; the main Automodel backend runs without them.

Installation paths and what each pins

The README describes three paths with different trade-offs:

• Source with uv is the recommended route and reproduces the actively tested stack: Python 3.13, PyTorch 2.12, and CUDA 13.2, with a CUDA 12 option. It installs into a local virtual environment and is the path the Dockerfile and CI use when run locked.
• The prebuilt NGC container (26.07.00) is the turnkey equivalent of that stack.
• pip/uv pip fallback is for people who already have their own Python/PyTorch/CUDA setup. It keeps that build and only requires Python 3.12+, PyTorch 2.7+, and an NVIDIA GPU for training. The authors warn not to use uv sync --locked on this path, because it replaces your stack with their pinned baseline.

A GPU is required for training and recommended for inference; CPU-only use is not described as a supported training setup. Installation, compatibility, and build details are in the official docs.

Security and version caution

PyTorch 2.6 and later default torch.load to weights-only loading. Some checkpoints may need TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD=1; the README says to do this only with trusted files, since loading from untrusted sources can permit arbitrary code execution. Weights published on Hugging Face are separate downloads with their own model cards, so check each model's terms and requirements rather than assuming the framework's licence covers them.

The repository is Apache-2.0 licensed, so you can use and modify it commercially under those terms. Its release history shows frequent checkpoint updates, which may matter if you need reproducibility across model versions.

Project source and documentation