SheepSeb/tinyworldformer

This project builds two small neural networks — a world model that predicts the next game frame from the current frame and an action, and an actor that chooses actions from the generated frame — and runs both on a Raspberry Pi Pico 2 W. On the board the real game never executes: the actor only sees frames the world model produced, so the game is played and learned "inside its own dream."

What it demonstrates

The world model is a looped transformer that reuses one block's weights across 1–4 passes; more loops raise accuracy at a cost in speed. Frames are split into 2×2 to 6×6 patches, and a decoding step places single sprites globally so exactly one ball or player appears. The author notes this decoding rule lifted Breakout exact-frame accuracy from 91.6% to 99.7% on held-out data (README).

Five games are included: Breakout, Doom-lite, Flappy Bird, Pong and Pac-Man lite. Exact-frame rates at two loops range from 83.6% (Doom-lite, whose 3D view makes single-pixel errors count whole frames wrong) to 100% (Flappy Bird). The author argues data accuracy is insufficient because an agent can exploit model mistakes, so a second check replays dream frames through the real game rules; over 5,000 dreamed steps it found 742 wrong game states in Pac-Man but 0–1 in the other checked games (checking the dream).

The actor uses Stream AC(λ), which learns from one transition at a time without replay buffer or target network, and the author reports exploit-driven fixes such as noise augmentation when Doom-lite's dream lost an enemy sprite (README).

Ways to run it

• Pico 2 W firmware: a no_std Rust build with int8 matrix weights and float32 embeddings, selecting a world model and agent at build time. Every documented game fits the board's 4 MB flash and 512 KiB SRAM, though Pac-Man comes close at 460 KiB RAM (on the chip).
• Browser: the same int8 engine compiled to WebAssembly, with no wasm-bindgen, handling dream, agent and on-device learning (browser build).
• Native host: the Rust crate can run a game in a terminal.

Training and export use Python with uv, and the repository documents a multi-stage pipeline: record transitions, train the world model, train the agent, evaluate the dream, then export to Rust (reproduce). The author states world-model training used an RTX 4060, while the agent can learn directly on the Pico (HN reply).

Conditions and open points

The goal is a compact demonstration, not a general game engine. The author told a commenter that applying this to something like an open world would likely be hard without hallucinations (HN reply). The README says frame rate on the Pico itself was not yet measured, and Doom-lite and Pac-Man accuracy remains below the simpler games. No license is declared in the repository metadata, which matters for anyone intending to reuse the code.