Researchers Find AI Agents Confuse Old and New Facts in Memory Even After Correct Retrieval

The Problem with Long-Term Memory

If an AI agent has long-term memory — say, it remembers six months of conversation with a user, or holds a database of facts about a client — contradictions are bound to accumulate: something true yesterday may not be true today. A team of researchers from Otto von Guericke University Magdeburg (Germany) and the University of Kerala (India) asked a simple question: if a memory-retrieval system surfaces both the old and the new version of a fact, can a language model actually pick the right one?

The short answer: not always, and the more clutter surrounding the correct fact, the worse it gets.

How They Tested It

The researchers used the FactConsolidation task from the MemoryAgentBench benchmark — a set of numbered statements where a later number indicates a more current version of a fact. The facts are synthetic and deliberately contradict common world knowledge (for instance, the "real" author of a play isn't the one a model would recall from its training data), so the test specifically measures whether the model relies on the provided context rather than its own internal knowledge.

Two models were tested — Qwen2.5-7B-Instruct and Qwen3-14B — running via Ollama on a university GPU server. Relevant statements were retrieved using BM25 (a classic text-ranking algorithm based on keyword matching), with retrieval volume varied between 64 and 512 tokens.

What They Found

The first surprising result: when the volume of retrieved data increased from 64 to 512 tokens, Qwen2.5's accuracy on a 455-statement history dropped from 45% to 32%. Intuitively, more context should mean a better chance of finding the right answer. But a closer check showed the needed, most current fact was present in the retrieved context in all 100 questions at every volume level. The drop in accuracy wasn't caused by missing information — it was caused by the model getting increasingly confused among the extra data and picking the wrong fact.

The researchers then narrowed the experiment to a cleaner test: 74 questions where a fact had an outdated version, comparing two equally sized sets of four statements. The first set contained the current fact plus three random, irrelevant statements. The second set was identical, except one random statement was replaced with an outdated version of the same fact (the current version remained in place either way).

The results were stark:

• Qwen2.5-7B answered all 74 questions correctly when surrounded only by irrelevant statements. But when one of those statements was replaced with an outdated fact, accuracy dropped to 33 correct answers out of 74 (44.6%).
• Qwen3-14B proved more robust: accuracy fell from 100% to 87.8% (9 errors out of 74).
• In both cases, every single error involved the model choosing the old, superseded fact — not random noise or hallucination.

In other words, an outdated fact of the same size proved far more dangerous to the model than simply irrelevant text — even though, intuitively, the model should have handled it fine, since the correct answer was visible in context and the rule ("a higher number takes precedence") was stated explicitly.

The effect held up on a larger memory history too — 2,310 statements and 100 questions: accuracy with narrow retrieval (64 tokens) was 69%, versus 58% with broad retrieval (512 tokens). Every error in this sample was also a case of picking the outdated fact.

Why It Matters

The key practical takeaway: you can't evaluate an AI agent's memory system with a single accuracy number. Two distinct questions need to be separated — did the retrieval system find the correct, current fact, and did the model actually select it instead of an older version, when both are present in context? Measuring only the final answer's accuracy risks conflating a retrieval problem with a model "reasoning" problem — and the fixes for each are different.

The researchers propose a concrete direction for memory-system developers: rather than retrieving all relevant facts and handing them to the model as-is, outdated entries should be explicitly flagged as inactive, or down-weighted during retrieval — while still preserving the change history for transparency. Otherwise, any system where memory evolves over time — a CRM's memory, a personal assistant, a corporate knowledge base — risks occasionally surfacing long-superseded information to users, even when the correct, current fact was technically "found."

A caveat: the study tested only two models from the Qwen family on a single synthetic benchmark with relatively simple update chains (no more than two versions of a fact). The authors acknowledge they didn't test the effect on other model families or on systems using vector-based (rather than lexical) retrieval — so this isn't yet a fully universal conclusion, but it's clearly a signal worth factoring into the design of agent memory systems.