Have a project in mind? Let's talk.

Memorilla is a memory system from Stanford's SNAP lab that lets a frozen LLM use an entire document collection without stuffing it into the context window. Released as open source with a paper, code, and trained checkpoints, it compresses everything the model might need into 16 query-dependent "memory tokens", and it beats full-context inference on knowledge-intensive tasks while using 108x fewer tokens. For anyone building agents that must remember their own history, it is the most concrete result this year on a problem most teams still solve with a vector database and hope. Source Source
The problem the paper names is simple: a frozen LLM can only use what fits in its context window, so access to past conversations, long documents, or an agent's earlier actions depends entirely on what a memory system places there. Memorilla's answer has three operations. A frozen encoder (Qwen3-Embedding-4B in the released recipes) maps each document chunk to an embedding once, producing a query-independent store that is computed a single time and persisted; only added or changed documents are ever re-encoded. For each question, a frozen query encoder embeds the query, and a trainable memory module reads the store and returns a fixed number of soft memory tokens in the frozen LLM's input embedding space. The decoder (a frozen Qwen3-8B) then answers from those tokens plus the query text. Source
The memory module itself is small: 194 million parameters in bfloat16, per the Hugging Face model card, configured for 16 memory tokens with no document self-attention. Because the LLM always receives exactly 16 tokens regardless of collection size, the design sidesteps the cost curve that makes long context expensive. The released checkpoints cover three general training stages plus fine-tunes for two personalization benchmarks, all under the Apache-2.0 license. Source Source
The headline number is FactKG, a knowledge-intensive question answering benchmark: Memorilla reaches 84.33% accuracy against 80.43% for full-context inference, meaning the compressed readout beat giving the model every document, while using 108x fewer tokens to represent the sources. Across eight benchmarks spanning personalization, long-document, and knowledge-intensive QA, the paper reports the best average performance against baselines that include Mem0 and MemoRAG, the two strongest named memory systems. On NarrativeQA, Memorilla compresses books and movie scripts by more than 4,000x. Source Source
The GitHub README's personalization example is the one that makes the mechanism vivid. A user has 107 earlier conversations with an assistant, about 32,000 tokens, and asks for breakfast ideas. Memorilla reads all of it into 16 memory tokens, roughly 2,000 times fewer, and answers with suggestions that avoid spiking blood sugar, because in one of those 107 conversations the user revealed a family history of type 2 diabetes. The user never repeats that fact, and the same frozen Qwen3-8B without the memory answers with a generic list. This is the difference between retrieval that finds keywords and a trained module that learns what the query actually needs. Source
The paper's agentic result is smaller but strategically important. In a TextWorld environment, where an agent's document collection is its own growing history of past turns, Memorilla reduces the agent's tendency to wander between rooms it has already explored and raises the average success rate from 18.6% with full-history in-context learning to 24.2%. The claim the authors make is continual learning: as an agent accumulates experience, the memory store grows, while the cost of using it stays flat because the decoder still only ever sees 16 tokens. Source
That is the property production agent systems need. Most agent memory today is either full-history replay, which degrades and gets expensive as the trajectory grows, or RAG over past turns, which retrieves text that matches the query rather than text that matters for the decision. A trained readout that stays constant-size as history grows is a third option, and it connects directly to the memory governance questions we covered in our MongoDB Atlas agent engine post. If you are evaluating how to test agents against these memory behaviors, our AgentEnv repeatable evaluations guide covers the harness side. Source
The engineering numbers are as strong as the benchmark numbers. On one H100, growing a collection from 1 to 512 documents, about 128K tokens, leaves Memorilla's response time at 0.74 seconds, while putting all documents in the prompt slows the same model from 0.75 to 19.9 seconds, 27x slower. Only the memory module's cost grows with the collection; the decoder's cost is fixed. Source
The repository also ships a memory cache for the genuinely large case. By default the module reprocesses all document embeddings per question, which takes 74 ms at one million documents. With the cache enabled, the cross-attention keys and values are computed once when documents are written, so a question costs 8.1 ms at one million documents and 33 ms at four million, roughly a billion tokens at 250 tokens per document; torch.compile brings the million-document case to 5.0 ms. The cache uses four times the embedding memory, 80 GiB for four million documents, and its FactKG accuracy stays within 0.06 points of the default path. Source
Memorilla is a research release, and the requirements read like one. Inference needs Linux with a CUDA GPU and Python 3.11; the recipes were run on NVIDIA H100 80GB cards, and the quickstart expects one 80GB GPU and downloads about 25 GB of models on first run. Installation is a git clone plus uv, with FlashAttention-2 pinned. Training the memory module from scratch across the three stages takes about 19 hours on 4 H100s and pulls roughly 310 GiB of data and precomputed embeddings, while evaluating a checkpoint on all eight benchmarks takes about 15 minutes on one GPU. The trained modules are on Hugging Face and load through the same MemoryVLLM interface the README demonstrates. Source Source
For teams watching agent memory rather than training it, the practical path is the personalization checkpoints, pmv2 and pv4, which are fine-tuned for exactly the multi-session assistant scenario. Watching how the memory layer behaves in production also matters as these systems graduate from demos, which is the territory of our AI agent observability checklist. Source
This section is our assessment, not the sources' claims. Memorilla's real contribution is reframing memory as a trained component rather than plumbing. RAG treats memory as a search problem and hopes the retriever's taste matches the task; Memorilla trains the taste, and the FactKG result, better than full context at 108x fewer tokens, is the strongest evidence yet that the hope was misplaced. The caveats are equally clear. It is bound to a frozen Qwen3 stack, so adopting it means accepting that decoder and encoder. It needs serious GPU hardware even for inference, which rules it out for most teams' near-term production use and makes it a research signal more than a deployable service. And the agentic result, while real, is a 5.6-point gain on TextWorld, a long way from reliable continual learning. Our expectation is that the idea travels before the code does: memory tokens as a fixed, learned interface between storage and a frozen model is a pattern other labs will adopt, and it is worth tracking precisely because it makes agent memory cheaper as history grows instead of more expensive.
Memorilla is an open-source memory system from Stanford SNAP that gives a frozen LLM access to an entire document collection through 16 query-dependent memory tokens. A frozen encoder embeds documents once, and a small trainable module retrieves what each query needs into the LLM's input embedding space. Source
It runs three operations: encoding, where a frozen Qwen3-Embedding-4B maps each document to a stored embedding; retrieval, where a 194M-parameter trainable module reads the store conditioned on the query; and generation, where a frozen Qwen3-8B answers from 16 memory tokens plus the query text. Source Source
On FactKG it scores 84.33% versus 80.43% for full-context inference while using 108x fewer tokens, and it reports the best average performance across eight benchmarks against baselines including Mem0 and MemoRAG. It compresses NarrativeQA books and scripts by over 4,000x. Source
Yes, within limits. In a TextWorld environment, feeding an agent's growing history through Memorilla raised average success from 18.6% to 24.2% and reduced wandering through already-explored rooms, because the decoder's cost stays fixed no matter how long the history gets. Source
Ready to take the first step towards unlocking opportunities, realizing goals, and embracing innovation? We're here and eager to connect.