Microsoft veteran runs a transformer on a 6 MHz PDP-11 with 64 KB of memory

Microsoft veteran Dave Plummer has shown that a transformer can be not only explained in simple terms but actually run on hardware from the late 1970s. His experiment with a PDP-11 minicomputer at 6 MHz and 64 KB of RAM brings the AI conversation down to earth: training is a lot of arithmetic, repetition and careful optimization.

An old computer, a new task

Plummer is known as a developer who helped build important components of Windows in the past. In his new video, he went for a demonstration of a basic principle behind modern models rather than a nostalgic stunt for likes. The experiment centers on a 47-year-old PDP-11 system, a machine from an era when nobody dreamed of large language models.

The contrast is what makes the project convincing: if a stripped-down transformer can be trained on such a device, the core of the idea is far simpler than it looks against the backdrop of data centers and billion-dollar budgets.

The PDP-11 ran Attention 11, a model written in PDP-11 assembly language by developer Damien Boureille. Its task was modest only at first glance: take a sequence of eight numbers and output it in reverse order. The point is not to memorize a few examples but to grasp a rule that will work on new input data. This is exactly what Plummer stresses: even in such a toy scenario, the model must learn structure rather than simply guess the next answer from a pattern.

How the model was squeezed down

To give the experiment any chance of working, the developers had to compress the architecture to the limit. Attention 11 is not a miniature copy of ChatGPT but a single-layer transformer with one attention mechanism, taken to a state of engineering minimalism. The model has just 1216 parameters. Instead of the memory arrays and accelerators typical of modern AI projects, it uses fixed-point computation, and the forward pass was reduced to 8-bit precision. In essence, it is a teaching skeleton of a transformer, keeping only what is needed to demonstrate the training process itself.

• 1216 parameters instead of billions
• fixed-point computation
• 8-bit precision for the forward pass
• optimization of nearly every processor cycle
• a task of discovering a rule, not memorizing examples

But even with such constraints, the result was not decorative. Plummer reported that the model reached 100% accuracy in roughly 350 training steps. On a PDP-11/44 with a cache board, this took about three and a half minutes. Compared with modern GPUs, that is of course museum-grade speed. But for a 6 MHz machine with 64 KB of RAM, the fact of full-fledged training matters more than the absolute time: the experiment proves that the principles of transformers require no magic, only resources and good engineering.

Not magic, but math

The main goal of the project was not to find a useful practical replacement for modern models. Plummer tried to show something less pleasant for romantics: there is no sacred fire at the foundation of AI. There is a cycle of errors, corrections and repetitions in which the weights gradually adjust to the task. His demonstration therefore works as an antidote to the mystification of neural networks. It strips away the marketing layer and leaves bare mechanics that can be watched almost frame by frame.

"From guessing to knowing."

That is how Plummer describes the moment when the model stops stumbling and starts consistently applying the rule it has found.

That is the most interesting effect of the experiment: the viewer sees not a finished smart answer but the birth of an ability through successive corrections. Against the backdrop of AGI talk, this is sobering. Modern systems impress not because they break the laws of computation, but because the same mechanism runs at colossal scale, on incomparably larger data, models and compute clusters.

What this means

The PDP-11 experiment does not prove that ChatGPT can be ported to a retro computer. It does, however, show something else very clearly: the basic ideas of transformers are compact enough to be understood, reproduced and trained even on ancient hardware. For the market, it is one more argument for efficient small models and careful optimization, especially now that the cost of compute is becoming a competitive factor in its own right.