Context engineering: why AI agents get lost in long conversations, and how developers are fixing it

Developers of conversational AI agents with RAG, tools and memory describe the same problem: the system works well in short conversations, but after a few dozen iterations the model starts confusing tools, pulling outdated calls into its answers and relying on an error that once made it into the context. A new discipline, context engineering, is emerging to address this.

How this differs from prompt engineering

The key point is that a new, better prompt does not always solve agent degradation. The issue isn't how the instruction is worded, but which information reaches the model before each next reasoning step. Prompt engineering works with the text of a single request, RAG with retrieving external data, and MCP with standardizing access to tools. Context engineering operates a level above: it manages the entire contents of the model's context window at every step of the conversation, including what goes in, what gets removed and in what order it is presented.

• The problem doesn't show up right away, but after a few dozen iterations of a conversation with the agent
• Symptoms: confusion when choosing tools, use of outdated calls, entrenchment of earlier errors
• Context engineering stands apart from prompt engineering, RAG and MCP as a separate discipline of context management

Why the agent starts making mistakes

As a conversation grows, old tool calls, intermediate results and facts that are no longer valid pile up in the model's context window. The model does not automatically distinguish fresh information from stale: it treats everything in the context as potentially relevant. If an error gets in once, the model may rely on it in later steps, widening the gap between its picture and the actual state of the task.

What this means

As AI agents move from demos to long production workflows with RAG, tools and memory, context management is becoming an engineering discipline of its own, on par with prompt design and model selection. Teams that build explicit rules for selecting and cleaning context before each step get more stable agents in long conversations.