karpathy/nanochat

What it is

nanochat is a deliberately small, readable harness for the full LLM lifecycle on a single GPU node: tokenization, pretraining, supervised finetuning, evaluation, and inference. Instead of a configurable framework, it exposes one complexity dial, --depth (the transformer's layer count), and derives the other hyperparameters (width, heads, learning-rate schedule, training horizon, weight decay) so the resulting model is compute-optimal. A depth of roughly 24–26 lands in GPT-2 capability territory.

The project's framing claim is cost: a GPT-2-level run described in the README fits in about two hours on an 8xH100 node, roughly $48 at ~$3/GPU/hr, and potentially closer to $15 on spot instances — versus the ~$43,000 cited for the 2019 GPT-2 training. GPU pricing and availability change, so treat those figures as the author's estimates, not a billed outcome.

Quick start

Dependencies are managed with uv. Pick the extra matching your hardware and activate the environment:

uv sync --extra gpu    # CUDA (A100/H100 etc.)
uv sync --extra cpu    # CPU-only / MPS
source .venv/bin/activate

To reproduce the reference GPT-2-grade run, start an 8xH100 machine (the README mentions Lambda as the author's provider) and run the speedrun script, ideally inside a screen session since it takes roughly 1.5 hours:

bash runs/speedrun.sh

When it finishes, chat with the trained model through the CLI (keep the uv environment active):

python -m scripts.chat_cli

Expect short, simple answers: the speedrun model is a ~4e19 FLOPs capability model, and the README's example dialogue shows it confidently mis-explaining Rayleigh scattering.

Constraints and practical notes

• Hardware. The reference pipeline assumes 8xH100. It also runs on 8xA100 (slower), and dropping torchrun lets it run on a single GPU with gradient accumulation, taking about 8x longer. GPUs with less than 80 GB VRAM require lowering --device-batch-size (default 32) to avoid OOM.
• CPU/MPS. runs/runcpu.sh is a toy-scale example that shrinks the model to fit a few tens of minutes; the README explicitly says results will not be strong.
• Precision. Precision is managed through a global COMPUTE_DTYPE rather than autocast: bf16 on SM 80+ CUDA, fp32 on older CUDA and CPU/MPS by default. NANOCHAT_DTYPE overrides it, and fp16 training enables a GradScaler.
• Scope. It is a strong baseline codebase, not an exhaustively configurable framework; changes are expected to hold across all depth settings. The project asks PR authors to disclose substantial LLM-written contributions.
• License. MIT.

Where to look next

For research iterations, the README points to the scaling-law and miniseries scripts, and a favorite quick loop is a ~5-minute 12-layer (--depth=12) pretraining run tracked in wandb (the command). Longer-form material lives in the project guides, including the write-up of the sub-$100 GPT-2 journey, and questions can go to the Discussions tab or the DeepWiki index.

HotRepo's take

nanochat appeared on the all-language GitHub daily ranking on 2026-11-02 at #14 with 325 stars gained that day, against a lifetime total of about 59,600 stars and 8,400 forks. That is a snapshot, not a trend line, so it says little about sustained momentum — but it does show that a small, opinionated training harness with a public speedrun leaderboard keeps drawing attention. The more substantive signal is the leaderboard itself: the README's recorded runs have compressed time-to-GPT-2 from 3.04 to 1.65 hours between January and March 2026, which is the kind of measurable community progress the repo is designed to incentivize.