Stanford PhD Derives the Transformer From Scratch, Starting With One Memory Problem

On October 15, a Stanford computer science PhD published a blog post with a provocative premise: if the paper "Attention Is All You Need" did not exist, could you derive the Transformer step by step from a single question, namely how to make a machine remember as much as possible and retrieve any item on request? Nine years after the original paper, the post has spread through Chinese tech media, including Jiqizhixin.

A Problem Instead of an Architecture

The Transformer is usually explained top-down: here is the attention block, here is the feed-forward layer, here are the positional encodings. The blog post works bottom-up. Its starting formulation does not contain the word "attention." It contains only memory and a requirement for random access. If you need to store a lot of information and reach any piece of it, a recurrent network with a single hidden state quickly hits a bottleneck. Every access has to pass through a compressed state, and the gradient has to travel through dozens of steps.

From there, the logic almost builds itself. You need keys that locate items and values that get returned. You need soft selection instead of hard selection, so that training can proceed by gradient. You need several independent "queries" to the memory at the same time, which is where multi-head attention comes from. You have to inject the order of elements explicitly, since recurrence is gone. The specific derivation steps in the blog cannot be confirmed from the available excerpt, but this is the route the framing itself suggests.

What Clues Were Already in the Literature

The thesis that "the keys were on the table" can be checked against dates. Long before 2017, all the necessary pieces had already been published:

• 2014, Bahdanau et al.: additive attention in seq2seq translation, where the decoder looks at all encoder states.
• 2014, Graves et al., Neural Turing Machines: external memory with content-based addressing, fully differentiable.
• 2014–2015, Memory Networks and End-to-End Memory Networks: soft memory access over several "hops."
• 2015, Luong et al.: simplified multiplicative (dot-product) attention.
• 2015–2016: residual connections (ResNet) and layer normalization, without which a deep stack does not train stably.

The original 2017 paper assembled these elements and removed recurrence entirely. Its eight authors reported 28.4 BLEU on WMT 2014 English-German and 41.8 BLEU on English-French. The large model trained for about 3.5 days on eight P100 GPUs, which is almost nothing by the standards of today's clusters.

Why Texts Like This Spread

The "could you have done it?" format has clear value. It turns an architecture from an artifact you have to memorize into a chain of engineering decisions, each answering a specific constraint. That is more useful than retelling the diagram from the paper: understanding the reasons helps when you need to modify a model rather than just run an off-the-shelf one. Hence the interest. The Transformer remains the foundation of almost all large language models, and any talk of replacements (state space models, linear attention, hybrids) requires understanding which problem the original attention was solving.

How to Use This in Practice

If you work with LLMs or are about to start, treat the blog's framing as an exercise. First, write down on paper what the memory should do: store pairs, search by similarity, return a weighted mixture. Then implement single-head attention in roughly 30–50 lines of PyTorch or NumPy and check that it works on a toy copying or lookup task. After that, add scaling by the square root of the key dimension, order via positional encodings, and multiple heads. At each step, remove a component and see what breaks.

Teams evaluating Transformer alternatives can run the same exercise on a candidate: which subproblem (addressing, training parallelism, context cost) does it solve better, and what does it give up in return? The blog post itself is worth reading in the original: only the introductory excerpt is available in the source.