logitcai/tinybrain
Machine LearningA from-scratch Chinese-first small language model project with an open dataset, tokenizer and full pretraining-to-RL pipeline in PyTorch.
Project overview
tinybrain is a research-style project that builds a Chinese-first small language model from scratch. Unlike many teaching repos that start from someone else's data, it also builds the data pipeline: 26 public sources are downloaded, cleaned, deduplicated across sources, and split into four token budgets (1B / 5B / 20B / 60B), published as the tinybrain dataset on ModelScope dataset page. A SentencePiece BPE tokenizer (32,768 vocabulary) is trained in the repo, and the model combines multi-stream residuals (mHC), Engram lookup memory, 3:1 sliding-window attention, optional MoE at the large preset, and the Muon optimizer. The training chain covers pretraining, annealing, SFT, domain experts, multi-teacher on-policy distillation, and RLVR/agent RL/RLAIF README.
It suits people who want a readable end-to-end Chinese LLM implementation, small experimenters with a single consumer GPU, and researchers interested in mHC / Engram / Dr. GRPO-style training details. It is not a competitive chat assistant: the author reports that the released RC1 weights still fail on arithmetic, and independent final evaluation is not done yet.
Quick start
Run a released Small chat model. Requirements: Python 3.11+, and enough disk for a 639 MiB weight file.
python -m pip install modelscope
modelscope download --model caiwangzsnm/TinyBrain-Small-Chat-20261119-RC1 --local_dir ./tinybrain-small
python -m pip install ./tinybrain-small/tinybrain-0.1.0-py3-none-any.whl safetensors
tinybrain chat ./tinybrain-small --device auto
Medium weights (4.99 亿 parameters, 1.86 GiB) are available on the Medium model page. To train from source instead, clone the repo, install with pip install -e ".[data]", download pretrain_nano.parquet from ModelScope, and run tinybrain pretrain then tinybrain sft; tinybrain generate and tinybrain chat --demo check the results, and tinybrain serve opens a local chat page at 127.0.0.1:8001 README quick start.
Architecture notes
The model is the main technical content. Residual connections are widened to four streams, mixed by a doubly-stochastic matrix enforced with Sinkhorn iterations; Engram retrieves 2- and 3-gram memory rows by hash and injects them through gating; attention is grouped-query with QK normalization, RoPE, and three sliding-window layers out of every four. The large preset switches to a 48-choose-8 MoE with two shared experts, bias-based load balancing, and an optional LatentMoE. Training uses Muon for matrices and Adam for embeddings, output head, memory tables and scalars model section.
Constraints
Smallest training presets are sized for a 16 GB consumer GPU: the author reports small with 2048×16 at about 70k tokens/s and ~12.5 GiB peak on an RTX 5070 Ti; medium fits with micro-batch 4 and gradient accumulation 4. CPU and MPS run ordinary PyTorch paths. CUDA acceleration relies on Liger Kernel and FlexAttention, and Windows requires a matching triton-windows and PYTHONUTF8=1. Multi-GPU (ZeRO-1) is supported only for pretraining and annealing, not SFT, distillation or RL. RLAIF and some evaluation need a separate Qwen3.8-27B judge server, which does not fit on a 16 GB card.
The code is MIT licensed, but the mixed dataset has no unified license: each document carries a license column, and Baidu Baike and shu are marked as unconfirmed, so commercial use needs per-source checking. The author also lists unfinished items: independent final evaluation for Small and Medium, rubric calibration, classical Chinese/poetry benchmarks, and no Large training yet README.
HotRepo's take
tinybrain entered HotRepo's new-project list on 2026-11-20 with 137 stars, adding 16 stars in a roughly one-hour window and 4 forks total; no all-language daily rank is recorded, so this is a small early signal rather than evidence of adoption. What stands out is the breadth relative to typical from-scratch repos: the data pipeline, tokenizer, mHC/Engram model code, and online distillation plus Dr. GRPO-style RL are all documented with concrete formulas and file paths, and the author explicitly reports weak RC1 accuracy and missing independent evaluation rather than overselling results.
Written by AI from the project's README, updated . The project's own docs are the source of truth.
You might also like
- pytorch/torchtitanA PyTorch-native training stack for pretraining, fine-tuning, and RL of large generative models across multi-GPU clusters.
- jingyaogong/minimindAn open-source, from-scratch PyTorch LLM training pipeline with a ~64M model and full pretrain-to-RL stages for learners on a single GPU.
- rohitg00/ai-engineering-from-scratchA 542-lesson curriculum that teaches you to implement, test and evaluate model internals, retrieval and agent runtimes yourself.
- Helldez/BigMoeOnEdgeRun MoE GGUF models larger than available memory by streaming experts from flash, with a local OpenAI-compatible server.