Models Learn on the Fly: One SGD Step Cuts Loss by −8% and Beats a 32k Context

If you let a language model take gradient steps on the text it is reading, it gets better at predicting the rest of the document. The technique is called dynamic evaluation, and it has been known for a long time. But it has barely been measured on hybrid architectures with linear attention, on diffusion language models, or on standard long-context benchmarks. Surya Saka measured it. On models with classic softmax attention, next-token loss drops by 5–8%. A model with a 2k-token window and updatable weights beats the same frozen model shown all 32k tokens. But on tasks that require finding a fact in a long document, there is no gain at all.

One SGD step per 128 tokens instead of a long context

The setup is minimal. It uses one SGD step per 128-token chunk, updates only the MLP output projections in the last quarter of the blocks, involves no prior training, and uses a 2k-token attention window. For infrastructure, one detail matters: the KV-cache for 2k tokens is roughly 16 times smaller than for 32k, and the gradient is computed only over the tail of the network. On Qwen3-1.7B, Qwen3-4B and Gemma-4-E4B, the loss gain was 5–8% per document.

A separate test covered information-retrieval tasks. The author ran RULER, BABILong, LongBench-v2, SCOTUS, HotpotQA, MuSiQue, ContractNLI and citation completion, with at least 100 examples per test and two model sizes. Weights updated on the full document did not recover a single point of the accuracy lost to context truncation. The author's conclusion: the adaptation tunes style and vocabulary and does not handle fact retrieval.

Hybrids already adapt; diffusion gains twice as much

On Gated-DeltaNet hybrid models Qwen3.5-4B and Qwen3.8-27B, the gain does not exceed 1% at any learning rate. The explanation is simple: linear-attention blocks update their own state as they read, so they already perform the same adaptation without an external gradient. An extra SGD step adds almost nothing here.

Masked diffusion models behave differently. The mechanism carries over to them, but only if the learning rate is lowered and the updates are clipped. The effect is then larger than on autoregressive models.

• Qwen3-1.7B, Qwen3-4B, Gemma-4-E4B (softmax attention): −5–8% loss, better than a frozen model with a 32k context.
• Qwen3.5-4B, Qwen3.8-27B (Gated-DeltaNet hybrids): under 1% at any learning rate.
• Nemotron-Diffusion-8B (masked diffusion): about 8%.
• DiffusionGemma-26B (masked diffusion): 17–20%.
• Configuration: one SGD step per 128 tokens, MLP projections in the last quarter of the blocks, 2k window.
• Benchmarks with no accuracy gain: RULER, BABILong, LongBench-v2, SCOTUS, HotpotQA, MuSiQue, ContractNLI, citation completion.

Facts are not written into the weights, but conclusions are

The naive idea goes like this: write a fact in with a gradient on the next-token loss, and the model will remember it. In practice, the model memorizes the prompt format. With chat prompting, it recalls 30% of facts, versus 80% when the facts are supplied in context. The cost is also high: on GSM8K, the model loses 36–38 points at any update dose.

The author proposes an alternative that writes a conclusion in as behavior. The model's own judgments, conditioned on the conclusion, are distilled into a low-rank adapter. Over 100 games, the result was 0.83 and 0.88, versus 0.84 and 0.88 for the same conclusion supplied in context. GSM8K does not change. The conclusion itself can come from an inquiry loop, a cycle in which the model looks for an answer by asking questions chosen by an information-gain criterion.

Where this fits in the inference stack

For teams running softmax models on long documents, dynamic evaluation may replace part of an expensive context: a short window plus a lightweight online update gives lower loss than 32k tokens with frozen weights. But it suits language modeling, stylization and work with narrow vocabulary, such as legal or medical text. For RAG and fact lookup in contracts, it is useless, and those tasks still need context or external memory.

For hybrid architectures, a separate SGD layer will not pay off. For diffusion models it looks promising, but it requires tuning the learning rate and clipping. If you need to fix a specific piece of knowledge in a model, distilling the conclusion into a LoRA adapter works better than writing facts in directly through the loss: it keeps quality at the in-context level and does not break math ability. The paper runs 19 pages with 13 figures and 12 tables, and is published on arXiv as 2610.33025.