snap-stanford/memorilla

What it does

Memorilla is a memory system for frozen LLMs. It reads an entire document collection, such as a user's past conversations or a long book, and distills the query-relevant information into a fixed, small set of memory tokens (16 in the released setup) that are fed to the unchanged model. Because the token budget does not grow with the corpus, the same decoder can be used whether the collection holds a few documents or hundreds.

The repository ships training and evaluation code, reference checkpoints, and benchmark recipes. The paper reports that on FactKG the memory tokens outperform giving the model every document (84.33% vs. 80.43%) while using 108x fewer tokens, and that on TextWorld the approach raises average success from 18.6% to 24.2% when the memory module learns from rewards. These are the authors' reported results; the repository lets you reproduce them or apply the method to your own data.

It targets researchers and engineers who work with long-context retrieval, personalization, or agent memory and who have CUDA GPUs available.

Quick start

The instructions require Linux with CUDA GPUs and Python 3.11; the recipes were run on 80 GB H100 cards installation steps. Install with uv:

git clone https://github.com/snap-stanford/memorilla.git && cd memorilla
uv venv --python 3.11 && source .venv/bin/activate
uv pip install -e .
uv pip install --no-build-isolation flash-attn==2.7.4.post1

FlashAttention-2 is used by default for training; evaluation does not need it, and training can fall back to PyTorch attention with --attn_implementation sdpa optional extras.

Then run the example inference call on one PersonaMem-v2 test item, which needs one 80 GB GPU and downloads about 25 GB of models and a 37 MB test set on first use getting started:

from memorilla.benchmarks import PERSONA_PROMPT
from memorilla.data import ensure_data, load_split
from memorilla.vllm import MemoryVLLM

row = load_split("pmv2", "test", ensure_data("pmv2", ["test"]))[73]
model = MemoryVLLM.from_pretrained("memorilla/Memorilla-Qwen3-8B", "pmv2")
answer = model.answer(row["question"], row["documents"], system_prompt=PERSONA_PROMPT)
print(answer)

A printed answer is the expected result: the model's response conditioned on the compressed memory rather than the full conversation history.

To train from scratch, run the staged configs (about 19 hours on 4 H100s and roughly 310 GiB of downloads for the three stages), or continue from released checkpoints downloaded with hf download memorilla/Memorilla-Qwen3-8B --local-dir checkpoints training and evaluation checkpoints. Evaluation of a checkpoint on all eight benchmarks takes about 15 minutes on one GPU.

Using your own data

Datasets are parquet files with one row per question and the columns collection_id, question, answer and documents, placed under my_data/data/my_task/; embeddings are computed on first use. The single_task recipe warm-starts from the Stage 2 checkpoint, and --init_memory "" trains from scratch data guide.

Constraints

• Hardware: Linux plus CUDA GPUs; the documented recipes assume 80 GB H100-class cards, and 4 GPUs for full training.
• The memory modules are tied to the Qwen3-8B decoder and Qwen3-Embedding-4B encoder used in training; the checkpoints are released with those defaults.
• Training and the full benchmark suite involve large data and embedding downloads (about 310 GiB for the three stages). Offline and multi-GPU issues are covered in the troubleshooting guide.
• PersonalizationV4 is released under CC BY 4.0, and each Memorilla-Data configuration keeps the license of its source dataset; the code is Apache-2.0 license.

HotRepo's take

The repository appeared on the Python daily list on 2026-11-10 with a best position of 15 and 62 stars gained that day, while a Hacker News thread from 2026-11-06 drew little discussion HN thread. The notable part is less the ranking than the positioning: a memory module that keeps the base LLM frozen and compresses an unbounded collection into a fixed token budget, with an accompanying paper, checkpoints, and a released personalization benchmark. Readers with the required GPU hardware can test the method directly; those without it can still use the paper and the data format documentation as a reference for long-context memory design.