DeepSeek shrinks its model's memory cache to 890 bytes per token, down from 3514

DeepSeek has posted DeepSeek-V4.1-Flash on Hugging Face and published a paper that focuses less on answer quality than on the cost of storing memory. When a language model reads a long text, it stores intermediate attention data for every token. That is the KV-cache. It exists so the model does not have to recompute everything from scratch each time it generates the next word. The longer the context, the heavier it gets. In V4.1-Flash, the global cache in HBM (a GPU's fastest memory) takes up 890 bytes per token. The previous V4-Flash needed 3514. For a session of one million tokens, that works out to roughly 0.9 GB instead of 3.5 GB.

The model is multimodal and built as an MoE (mixture of experts): only a subset of experts runs for each token. Its main trunk has 552 billion parameters. Another 196 billion sit in Engram memory tables. The context is designed for 1M tokens. The paper has around 500 authors. There is no training code, but the weights are published.

Three techniques that made serving cheaper

The main architectural novelty is a causal encoder-decoder design, abbreviated CED. Almost all top open models are decoder-only, where a single stack of layers both reads the prompt and writes the answer. Here the stack is split. During prefill, when the model reads the prompt, 8 billion parameters are active. During decoding, that is, while generating the answer, 16 billion are. It is a relative of the YOCO idea, in which the cache is computed once and then reused.

The second technique is the CSA2 attention mechanism. It takes the cache in FP4 format, meaning it stores numbers in just four bits, and reuses it across layers. Previously, each layer kept its own copy.

The third technique is SWA Bounded Replay. Sliding-window attention looks only at the most recent tokens, and its local cache previously had to be written to disk so a session could be continued later. Now it is not written: when a session resumes, a bounded fragment is recomputed. This costs compute, but it takes load off the disk.

The memory results:

• HBM cache: 890 bytes per token instead of 3514, almost 4 times less;
• long-term cache on SSD: roughly 1/8 of the volume of V4-Flash;
• active parameters: 8 billion during prefill and 16 billion during decoding.

What the benchmarks showed

Savings alone are not enough if the model got dumber. So the authors compared it with its predecessor and with the closed Opus-5 on agentic tasks, where the model works with code and a terminal on its own.

• DeepSWE v1.1 is a set of software engineering tasks: the agent edits code to fix a specific problem. V4.1-Flash solves 74.2%. V4-Flash managed 54.4%, and Opus-5 74.0%.
• Terminal-Bench 2.1 covers command-line tasks. Its Pass@1, the share of tasks solved on the first attempt, is 90.6% for V4.1-Flash. V4-Flash scored 82.7% and Opus-5 89.1%.

The gain of almost 20 percentage points on DeepSWE is substantial. On both tests the open model came out ahead of the closed one, although on DeepSWE the 0.2-point gap is within ordinary noise.

Where questions remain

The practical point of the work is that agent sessions of a million tokens are now limited by compute rather than by the amount of HBM or SSD needed for the cache. A long session with an agent that spends weeks accumulating a history of edits and terminal logs becomes cheaper. More such sessions can run on a single card at the same time. For companies that host models themselves, this means direct savings on hardware.

There are caveats, however. The authors did not show component-level ablations, meaning they did not switch the techniques off one at a time, so it is unclear which of the three delivers most of the savings. There are also no curves of context retrieval quality across the full million tokens. The claimed "1M window" is less well supported than one would like. On the hard scientific tasks of Terminal-Bench 4.0, the model falls clearly behind: 31.2% versus 51.8% for Opus-5.

Of separate interest is that V4.1-Flash appears to be the only large open model to have moved away from pure decoder-only. Before this, such a move was seen mostly in T5Gemma, with its encoder-decoder architecture. If independent measurements confirm the gain, other labs may adopt the split of the stack into reading and generation.