Google taught an LLM to “sleep”: SLEEP consolidates knowledge into weights, up to 10 million tokens

Google Research has published SLEEP, a framework for continual learning in language models. It separates the intake of new data from its consolidation in time, much as wakefulness and sleep work in humans. During “sleep,” knowledge moves from temporary memory into the weights of additional MoE blocks. In experiments, the team showed operation on sequences of up to 10 million tokens on BABILong, almost no quality drop when training sequentially on translation, and 79.2% on AIME 2024 against 69% on AIME 2025.

Two holes in a model's memory

Continual learning has two long-standing problems. The first is catastrophic forgetting: a model absorbs new information and loses skills it had mastered earlier. Gradient updates for the new task overwrite weights that matter for the old ones.

The second problem is that knowledge in the context window does not turn into permanent memory on its own. While the information sits in the prompt, the model uses it. Once the context ends, it is gone. Long context is also expensive: it has to be re-processed and held in working memory for every request. If knowledge is written into the parameters, it no longer takes up context and is available without being fed in again.

Wakefulness, sleep and “dreams”

SLEEP alternates between two modes.

• Wakefulness. The model works with incoming data and stores information in short-term memory.
• Sleep. Intake of new data is paused. The model activates additional MoE memory blocks and transfers knowledge into their weights, so that already consolidated information is disturbed less. After the transfer, the temporary memory is freed.

A separate stage is called “dreaming.” Random combinations of experts generate training examples, and the system selects the most useful ones and uses them for further training. In essence, the model prepares its own consolidation material, without an external dataset for each cycle.

One important caveat from the description: “sleep” here is a controlled fine-tuning cycle that follows rules set by the researchers. The model's architecture and code stay fixed. This is not a self-evolving organism but a regulated procedure on top of an ordinary MoE model.

What the experiments showed

• BABILong. The model works with information from sequences of up to 10 million tokens by consolidating it in parameters rather than holding it in context.
• Sequential translation training. Almost no drop in results on previously learned languages after learning new ones.
• AIME 2024. Accuracy of 79.2%.
• AIME 2025. Accuracy of 69%.

The translation result is probably the most telling. Languages are learned one after another, which is the classic testbed for forgetting: quality on the first languages usually sags most visibly there. The math results show that consolidating new knowledge does not destroy the ability to reason. Note also that the AIME 2025 result is lower than the 2024 one. The gap between the two years is the same one we see in many models, and it is too early to draw conclusions about the causes from a single publication.

Where this runs into practice

For engineers, the main question is cost. Moving knowledge into weights means extra fine-tuning cycles, which is compute that has to be planned for: while the model “sleeps,” it does not serve requests. Teams will have to decide how often to sleep, at what volume of accumulated memory, and how many MoE blocks to allocate for storage. These parameters are set by the researchers' rules, and teams will have to tune them for their own workload.

The potential gain lies in the economics of context. If corporate knowledge, long conversations or a continuously updated document base are consolidated in the weights, there is no need to carry millions of tokens in every request and pay to process them again. For agents with a long working history, that is a direct saving in memory and latency.

There is also a risk that the description does not remove: writing to weights is irreversible and harder to audit than a document in context or in an external database. Regulated industries will need to work out separately how to delete and verify consolidated knowledge. For now, SLEEP is best treated as a strong research result and a good candidate for testing in projects with continuously updated data. It is not yet a finished production component.