pytorch/torchtitan
Machine LearningA PyTorch-native training stack for pretraining, fine-tuning, and RL of large generative models across multi-GPU clusters.
What it is
torchtitan is a PyTorch-native platform for training generative AI models at scale, covering pretraining, supervised fine-tuning, LoRA, and reinforcement learning in one codebase README. It is maintained by the PyTorch project and targets researchers and engineers who train or study large models rather than application developers calling hosted APIs.
The central idea is composable parallelism: models declare how they shard, and FSDP2, tensor, pipeline, context, and expert parallelism stack into N-D layouts via DeviceMesh and DTensor. Every run is expressed as a Python recipe, so changes to a model or training setup are readable and copyable instead of hidden behind long flag lists config guide, recipes.
Built-in model implementations include Llama 3, Qwen3, DeepSeek V3/V4, gpt-oss, Kimi, Muse, and FLUX text-to-image, with debug configurations exercised in CI and verified recipes run in convergence and performance workflows. A companion RL stack, TitanRL, reuses the same model definitions and kernels in both the trainer and vLLM generation, including a batch-invariant mode for bitwise trainer/generator parity RL docs, parity note.
Quick start
The documented path installs from source with a PyTorch nightly on Python 3.11+, then runs a small debug model before moving to a real model.
git clone https://github.com/pytorch/torchtitan && cd torchtitan
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu132
pip install -r requirements.txt
NGPU=1 ./run_train.sh
Replace cu132 with your CUDA or ROCm build. The run_train.sh run uses the tokenizer bundled with the tests and trains a Llama 3 debug model for 10 steps, so no download is required quick start.
For a real model, download the tokenizer (Llama weights require access on Hugging Face), then pick a recipe by module and function name:
python scripts/download_hf_assets.py --repo_id meta-llama/Llama-3.1-8B --assets tokenizer --hf_token=<your-token>
MODULE=torchtitan_recipes.models.llama3 CONFIG=llama3_8b ./run_train.sh
The 8B recipe is documented as an 8-GPU run. To change a run, write a Python function that returns a modified config and point MODULE/CONFIG at it, as shown in the recipe example recipes.
Constraints and requirements
The mainline path expects recent PyTorch nightlies; the README explicitly warns that latest features require a recent nightly. Nightly and stable pip/conda installs exist, and each stable release pins the torch and torchao nightlies it was validated against releases.
The codebase is under active development, so APIs and recipes should be expected to move. Low-precision MXFP8/NVFP4 training needs additional torchao nightly pins installed separately; RL needs its own environment script that installs vLLM, Monarch, TorchStore, and a FlashAttention build. Multi-node Slurm runs require adjusting both the #SBATCH header and the torchrun --nnodes value, plus --nproc_per_node if nodes have fewer than 8 GPUs.
Real-model training depends on external model weights and their licenses. Source code is BSD-3-Clause, but third-party data and model terms remain the user's responsibility license.
HotRepo's take
On the 2026-11-19 daily Python-language ranking, torchtitan reached position 12 with no net star change recorded that day. That is a placement rather than an upward trend: the available record is a single snapshot, and a project at roughly 5.9k stars with over a thousand forks reads as an established training foundation rather than a breakout newcomer. The more telling signal for readers is not the ranking but the ecosystem around it—an ICLR 2025 paper, an AMD-optimized fork, SkyPilot integration, and an elastically deterministic data-loading integration from DatologyAI—which suggests adoption in real large-scale training workflows rather than tutorial use.
Written by AI from the project's README, updated . The project's own docs are the source of truth.
Hacker News
You might also like
- NVIDIA-NeMo/SpeechA PyTorch framework for building, fine-tuning, and running speech recognition, synthesis, and speech-LLM models.
- jingyaogong/minimindAn open-source, from-scratch PyTorch LLM training pipeline with a ~64M model and full pretrain-to-RL stages for learners on a single GPU.
- drost-ai/tinker-ai-skillTurns your Claude Code session transcripts into a fine-tuned LoRA model runnable locally via llama.cpp.
- pollen-robotics/microduck_rlTraining environments and sim2real recipes for teaching a small bipedal robot to walk with reinforcement learning.
- google-pytorch/torch_tpuLets PyTorch users run and train their models on Google TPUs through a native ATen backend.