Find your next open-source project

Search by name, purpose, or keyword — or paste a GitHub repository URL
Add a star history chart too →

jingyaogong/minimind

Machine Learning

An open-source, from-scratch PyTorch LLM training pipeline with a ~64M model and full pretrain-to-RL stages for learners on a single GPU.

Best position on GitHub Trending (daily): #13. Click to get this badge for your README
66,422
8,716
Python
Apache-2.0
2 years old

What it is

MiniMind is a from-scratch implementation of a small language model and its complete training pipeline, written in native PyTorch without relying on high-level abstractions from transformers / trl / peft (project overview). The mainline minimind-3 model is a 64M-parameter dense decoder-only Transformer aligned with the Qwen3 ecosystem, with a 198M-A64M MoE variant; both are released (model collection).

The repository covers tokenizer training, pretraining, supervised fine-tuning, LoRA, white/black-box distillation, RLHF via DPO, RLAIF via PPO / GRPO / CISPO, tool calling, agentic RL and adaptive thinking. All core algorithms are implemented by hand, and the project doubles as a tutorial for people who want to read and modify every part of the LLM stack.

Unlike fine-tuning an existing large model with LoRA, this project starts from an untrained network and small curated datasets, so a single consumer GPU can run the whole lifecycle. The author reports that SFT for one epoch on a single RTX 3090 takes about 2 hours, and the full mini pretrain + SFT path about 2.31 hours at roughly 3 RMB of rented GPU time.

Quick start: inference with a released model

Clone the repository and install dependencies (Python >= 3.10 is used in the author's setup):

git clone --depth 1 https://github.com/jingyaogong/minimind
cd minimind && pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple

Download a converted Transformers model, either from ModelScope or Hugging Face:

modelscope download --model gongjy/minimind-3 --local_dir ./minimind-3
# or
git clone https://huggingface.co/jingyaogong/minimind-3

Then run the CLI evaluation script from the repository root:

python eval_llm.py --load_from ./minimind-3

To try the Streamlit chat UI, copy the model folder under scripts/ first, because the demo scans that directory for weights:

cp -r minimind-3 ./scripts/minimind-3
cd scripts && streamlit run web_demo.py

Prebuilt Ollama, vLLM and llama.cpp paths are also documented; for example ollama run jingyaogong/minimind-3.

Quick start: training from scratch

Download the mini datasets into ./dataset: pretrain_t2t_mini.jsonl and sft_t2t_mini.jsonl from the dataset release. Then verify that your PyTorch build can see a GPU, and run the two mandatory stages from the trainer directory:

cd trainer
python train_pretrain.py
python train_full_sft.py

Outputs are written to out/pretrain_*.pth and out/full_sft_*.pth, with the wildcard standing for the model dimension (768 by default). Test the result with python eval_llm.py --weight full_sft, or evaluate the pretrain-only checkpoint with --weight pretrain. All training scripts accept --from_resume 1 to continue from checkpoints, and multi-GPU runs use torchrun --nproc_per_node N train_xxx.py. Optional stages such as LoRA, DPO, PPO, GRPO and agentic RL use their own scripts under trainer/.

Constraints and caveats

  • The models are tiny by design. The released 64M model has basic conversational ability but limited factual knowledge and generalization; the README itself notes that a 0.1B-parameter model suffers from severe reward sparsity on general reasoning tasks.
  • Training needs a working CUDA (or CPU/MPS) PyTorch environment; CPU training is possible but much slower, and multi-GPU support assumes DDP or DeepSpeed.
  • The shipped tokenizer has a 6,400-token vocabulary, which is more compact but less efficient than Qwen-style tokenizers; the README advises against retraining it to preserve compatibility.
  • Tool calling is trained on about 10 simulated tools with roughly 100k synthetic samples, so generalization to unseen tools is limited, and explicit thinking is not yet stable when combined with tool calls.
  • Weights for every experimental stage (DPO, PPO, GRPO, CISPO, agent, LoRA) are not all published and maintained; some are kept for verification only.
  • The project is Apache-2.0 licensed.

HotRepo's take

On 2026-11-17 the repository appeared on GitHub's all-language daily list at position 14 with 188 stars added that day, against 65,623 total stars and 8,605 forks. Earlier records show back-to-back days with about 1,005 stars gained (2026-09-01 and 2026-09-02), so interest is not a one-day spike. An HN thread in March 2026 described it as a clean resource for understanding the modern LLM stack, covering tokenizer training through SFT and LoRA (HN comment). That comment described a ~25M model, which reflects an earlier README revision rather than the current 64M release, so use the current repository as the reference. For readers who want runnable weights and a complete training loop at personal-GPU scale, the combination of published datasets, released checkpoints and hand-written algorithms is the main draw.

Written by AI from the project's README, updated . The project's own docs are the source of truth.

Hacker News

MiniMind: End-to-end GPT-style LLM training pipeline in pure PyTorch
▲ 3 · 1 comment · March 24, 2026

You might also like

  1. karpathy/nanochatA minimal, hackable codebase for training a GPT-2-class LLM end to end on a single GPU node.Python59,586
  2. unslothai/unslothA desktop app to run and fine-tune LLMs and diffusion models locally, aimed at people who want training without a cloud GPU.Python78,537
  3. MakazhanAlpamys/SoupCLI that fine-tunes and post-trains LLMs from a single YAML, with layer streaming for small GPUs.Python10,061
  4. huawei-csl/MACaQQuantize text, image, video and audio models to low precision and run them with llama.cpp, vLLM, Diffusers or PyTorch.C++112
  5. NVIDIA/Model-OptimizerCompress LLMs, VLMs, and diffusion models with quantization, pruning, distillation, and speculative decoding for faster inference.Python5,531