Interactive-Active-AI/NextTransformer

Overview

NextTransformer is a PyTorch framework and benchmark for comparing Transformer alternatives under a fixed training budget. Instead of letting parameter count, memory or speed decide the comparison, every architecture is trained on the same data and tokens with theoretical training compute matched to within a tolerance, then evaluated at identical checkpoints. The included model zoo covers baseline dense attention, residual-stream variants, looped/recurrent-depth models, sparse attention, linear attention and hybrids such as GDN–MHA and GDN–SWA. It is aimed at researchers and engineers who want to test an attention or architecture idea against strong baselines rather than reimplementing a training stack.

The project also ships a replayable leaderboard and an observatory with learning curves, CORE task results, architecture diagrams and model cards. An arena mechanism lets an LLM or research agent submit an architecture under the same rules.

Protocol and scope

The comparison protocol pins the dataset (ClimbMix at a fixed revision), tokenizer, 2,048-token context and sample order. A run covers 49.152B input tokens in 46,875 updates of 1,048,576 tokens with seed 42. Compute is set at 4.672 GFLOPs per input token and audited at ±1% for admission and ±2% over the run. Evaluation combines validation BPB at 189 points, full CORE-22 at six checkpoints and an independent final BPB. Parameters, memory and speed are outcomes, not controls; the protocol documentation states the rules and the compute accounting explains what is counted.

Quick start

Prerequisites are Linux, NVIDIA GPUs and uv. Install and prepare data as follows:

git clone https://github.com/Interactive-Active-AI/NextTransformer.git && cd NextTransformer
uv sync --frozen --extra gpu --extra upstream && source .venv/bin/activate
uv pip install --no-deps --require-hashes -r requirements/linear.txt

cp site.native.example.json site.local.json   # set runtime_dir, resources and data_dir
hf download karpathy/climbmix-400b-shuffle --repo-type dataset \
  --revision 915333b4f8b8684f39aeaafea600fea6f43fb703 --local-dir /path/to/climbmix-400b-shuffle
source launch/env.sh --site-config site.local.json
python -m nexttransformer.prepare --site-config site.local.json

The dataset is about 600 GB. Then train the dense baseline on one 8-GPU node:

launch/train.sh --config configs/baseline/dense_full.json --site-config site.local.json \
  --output runs/dense_full --micro-batch 8 --formal --resume auto

For multi-node runs, install accelerate==1.15.0, export FLA_DISABLE_BACKEND_DISPATCH=1, and launch the per-node entry point with accelerate launch --config_file launch/accelerate.yaml, mapping node rank, main address and port to NODE_RANK, MASTER_ADDR and MASTER_PORT; on Slurm, submit launch/slurm.sbatch. Evaluation runs on schedule during training. The quick start guide covers smoke tests, evaluation, auditing and exporting a result.

Adding an architecture

A new method is a plugin folder under nexttransformer/methods/<name>/ containing method.json and a few hooks, with no changes to shared files. It must be calibrated to the shared compute budget with python -m nexttransformer.protocol.calibrate --methods <name> ..., then admitted through a pull request; a formal training run and result submission follow in a second pull request. See the plugin guide and CONTRIBUTING.md.

Constraints and limitations

The advertised run is a roughly 1.4B-parameter model trained for 49B tokens in 2–4 days on one 8-GPU node, so a full comparison requires substantial GPU and storage resources. The pinned dataset download is large. The arena is under active development; Arena v2, with MCP tools, metered compute and research traces, is scheduled for November 14, 2026. Checkpoints and the tokenizer are planned for release on Hugging Face but are not yet listed as available. The project is MIT licensed and builds on nanochat, modded-nanogpt and flash-linear-attention kernels.

HotRepo's take

NextTransformer was created on 2026-10-31 and first appears in HotRepo's new-project list on 2026-11-02, with 113 stars, 8 forks and 7 stars added in the recorded growth window. The scale of the claims — dozens of completed runs, audited compute matching and a multi-node training stack — far exceeds what three hours of star sampling can verify. The useful observation is structural: the repository couples a reproducible protocol, a plugin mechanism and a public leaderboard, which is the part worth evaluating. It has not yet recorded a GitHub all-language daily rank, so no trend conclusion is warranted from this single snapshot.