lyramakesmusic/neuron-explorer

Adopt it if you are already doing activation-steering research on a small open-weights model and have GPU time to spare; treat it as a research instrument, not a packaged product.

The problem

You want to know what an individual MLP neuron does, not just what a whole direction at layer 22 does. The usual loop is ad hoc: write a script that edits one neuron, run a few prompts, squint at generations, then throw the script away when you move on. Because nothing is archived per neuron and per dose, you cannot compare neuron 3501's three prompts against neuron 12000's six prompts two days later, and you have no baseline showing which effects are just noise.

This project changes that loop into a batch sweep plus a browser. Every MLP neuron in a selected layer is pushed both up and down on the same small prompt set, every rollout is written to disk with text, token ids, logprobs and top-k, and the explorer then lays those runs out over a 3D map of the model's layers. You get persistence and visual navigation instead of a one-off prompt.

What's different

The mechanism is the pairing of a full sweep with a control group and a downstream lens. sweep/rank.py scores units by distinctive words and cluster mass against 256 random-direction controls, and sweep/steer_lens.py records next-token logprob changes and KL for ±0.8 pushes on every neuron. That control comparison is the useful part: it gives you a way to tell a neuron with a repeatable effect from one that merely produced a flavourful sentence, without hand-reading 30k outputs.

The doses are fractions of the median residual norm leaving the layer, not raw activation deltas, so a +0.8 push is expressed relative to that layer's own scale. The README describes fixed prompt/dose grids such as {who, snake, orange} at −0.7 and +0.5/+0.8 plus an unsteered baseline. The trade-off is that this is a dense, GPU-heavy sweep: every neuron times several prompts times several doses, with the steering fork of vLLM required. The explorer's live chat then loads Qwen3.5-4B locally for steering, lens and ablation work.

One thing to understand before investing: the sweeps, stats and cache files are inputs to the explorer. The shipped caches only cover what the author generated; your own layers need your own sweep runs in runs/ and stats/. There is also a Claude panel that runs the claude CLI with the explorer as an MCP server, which reads neurons and can run the live model, move the map and use file tools and a shell in your home folder. That is a research assistant convenience, not a requirement for browsing existing runs.

Who should try it

It fits a mechanistic-interpretability researcher or a curious engineer who already has a CUDA pod, is comfortable with vLLM and Hugging Face internals, and wants to run dense neuron-level steering experiments on Qwen3.5-4B specifically. If you need to inspect one interesting neuron interactively, the three-column steered chat and per-neuron rollouts will save you from rebuilding that harness.

It is a poor fit for users who want a general-purpose interpretability tool for arbitrary models, or who expect to inspect effects without running the sweep. The explorer is built around Qwen3.5-4B and around data produced by these scripts; the README gives no model-agnostic path and no hosted version.

Requirements and limitations

The sweep path is explicit about its environment: a CUDA 13 GPU pod, the RhizoNymph/vllm steering fork, and setup_pod.sh writing an environment file. The explorer can load the model on MPS, CUDA or CPU, but the sweep is the heavy part. The repo declares no license, so check whether its terms suit your use before building on it.

The Claude panel has its own setup: the claude CLI must be installed and logged in, and the server description says it answers only pages and tools on this machine. The agent has file tools and a shell in your home folder, so treat that panel as code execution with access to your local files, not as a sandboxed chat. Ongoing upkeep is less about install and more about the research workflow: keeping runs, stats and cache directories consistent as you add layers, which the README only sketches through merge_runs.py and the depth report.

HotRepo's take

HotRepo first tracked this repository on 2026-10-24 and observed it on the NEW-REPOSITORY board (python, not the overall Trending board) at #22 with 673 total stars on 2026-10-26. That is one dated snapshot of a four-day-old repo, not a daily growth series, so persistence cannot yet be judged. Before spending GPU hours, verify the one thing that determines fit: whether the sweep scripts accept the layer and stats layout your own data needs.

Project source and documentation