Every AI Layer Is a Memory Store: A Plain-English Guide

Explainer · How AI models remember

A new essay by researcher Alex Wang argues that every layer of a transformer can be read as a memory store. Here is the idea in plain English, and why it matters if you build on AI.

The Short Answer

On 15 October 2026, Alex Wang published “You could have invented transformers from memory” on his site, alexwang.ai. Wang did his PhD in computer science at Stanford, working on long-context architectures, and was most recently a research scientist at Cartesia, according to his About page. He is not Scale AI’s Alexandr Wang.

His argument: a linear layer, an MLP and self-attention are three kinds of memory store. Each one holds key-value pairs and recalls a value by comparing a query with the stored keys.

What separates them is how well they recall once many memories are stored. In the X thread announcing the essay, he summed it up: “every layer is a memory layer.”

What a Memory Store Is

Wang defines a memory as one key-value pair. The key is used for lookup by similarity. The value is what you want back.

A good store is judged by recall, not by size. As he puts it, even infinite storage is “not very helpful if we can’t recall them when we need to.”

The problem is interference. The more memories you store, the more they get in each other’s way when you try to pull one back out.

The Simplest Store: A Linear Layer

The most basic store is a linear layer that adds up memories as outer products. Querying it compares the input with every stored key and returns a blend of the values, weighted by similarity.

Wang splits every recall into two parts: the target memory’s value (the signal) and a little of every other memory with a similar key (the interference).

For random keys, he estimates that the ratio of interference to signal grows like the square root of M divided by d_k. M is the number of memories and d_k is the key’s dimension. Recall fails once that ratio reaches 1, so a linear store holds about d_k memories.

He also shows that solving for the weights with linear regression, instead of just adding memories up, about doubles the capacity to 2d_k. That store recalls exactly up to d_k memories.

One aside in the essay links this to training. A single gradient-descent step on a linear layer has the same form as writing a new memory into it. Wang points readers to Irie, Csordás and Schmidhuber for more on that view.

The MLP: Mute the Weak Matches

A linear store weighs every memory, even ones that barely match the query. Wang’s fix is to pass each similarity through a nonlinear function first, so weak matches count for less.

That, he writes, is a two-layer MLP: the rows of the first weight matrix act as keys and the columns of the second act as values. He credits Sukhbaatar et al. and Geva et al. with earlier work on this resemblance.

Squaring the similarity does a lot of the work. In his example, a similarity of 0.1 becomes 0.01, while a similarity of 1 stays 1. On his estimates at d_k = 64, a squared-ReLU MLP holds about 2,700 memories, about 40 times the linear store, and a SwiGLU MLP about 2,900.

Wang also notes that the original transformer paper made the same link, but only in versions 2 and 3 on arXiv. There the authors called the feedforward layer “a kind of attention.”

Self-Attention: Use Every Power

Higher powers suppress weak matches even more. The exponential is a sum of every power, and normalising it gives softmax. “This is self-attention,” Wang writes.

With the scaling transformers use, he estimates the capacity at d_k = 64 at about 1.2 million memories, nearly 20,000 times the linear store. A longer run of his experiment gave about the same figure.

There is a key difference. An MLP’s keys and values are trained weights. Self-attention computes them from the tokens in its context, so each token stores one memory and the store grows with the context. Wang calls this non-parametric.

For long contexts, he describes a trick with several versions: raise the sharpness of the softmax as the number of memories grows, scaling with log M. He cites Su (2021), Scalable-Softmax, Qwen’s LogN-Scaling and the YaRN scaling in Qwen2.5-1M. In his experiments, that allows “essentially an unbounded number of memories with distinct keys.”

What the Estimate Does Not Measure

Wang is explicit about the limits. His capacity estimate counts only how many random vectors a single layer can recall before interference dominates the signal. It does not consider the training data at all.

For trained models, he points to other measures. He cites Morris et al., who estimate that GPT-style models memorize about 3.6 bits per parameter, and Allen-Zhu and Li, who find about 2 bits of factual knowledge per parameter.

Why It Matters for Builders

As we read it, the essay gives business readers a simple map of where a model keeps what it knows. The MLP weights hold what was learned in training. The context window is a working memory that is filled fresh with every request.

That split lines up with a choice every AI product team faces. You can rely on what the model already knows, or you can put your own documents, history and tool results into the context. On Wang’s account, attention is the layer built to recall from that second store.

It also helps explain why long context is an engineering problem, not just a bigger number on a spec sheet. Recall has to stay sharp as the context grows, which is why the scaling tricks he describes exist.

For strategy, the context is where a company’s own data meets a general model. As we read it, that is part of why proprietary context and the traces an AI product generates can matter as much as access to the model itself. That is our framing, not Wang’s claim.

For a shorter primer on attention, see our explainer on transformers and attention. For how assistants carry memory between chats, see agent memory explained, and for the hardware side, the memory wall.

The Bottom Line

Alex Wang’s 15 October essay reads a linear layer, an MLP and self-attention as three memory stores. On his estimates at d_k = 64, a linear store holds about 64 random memories before recall fails, a squared-ReLU MLP about 2,700 and self-attention with the standard scaling about 1.2 million. He stresses that this counts random vectors in a single layer, not what a trained model actually stores.

A note on sourcing. This explainer is built on Alex Wang’s essay, read in full on 20 October 2026, and his About page. Figures and formulas are his estimates for random keys and values; the diagrams are FourWeekMBA’s, and the second is a schematic, not data. The “Why It Matters for Builders” section is our reading, not his.