Why ChatGPT Forgets: How Language Model Context Windows Work

An hour into a session with ChatGPT, the model suddenly forgets details from your first messages. It contradicts itself and asks again about things you already settled. It hasn't malfunctioned. This is how the context window works, and anyone who uses AI for work should understand the mechanism.

What is a context window

A language model doesn't "remember" a conversation the way a person does. It processes text as a single block called the context. The context window is the maximum amount of text the model can take into account in a single request.

Context is measured in tokens. A token is roughly 3–4 characters of English or 1–2 words of Russian.

Window sizes vary across current models:

• GPT-4o — up to 128 000 tokens (about 96 000 words)
• Claude 3.7 Sonnet — up to 200 000 tokens
• Gemini 1.5 Pro — up to 2 000 000 tokens
• Llama 3 — from 8 000 to 128 000 tokens, depending on the version

Even 128 000 tokens comes to about 300 pages of text. That sounds like a lot, but in real work sessions, with a codebase, documents, and a long back-and-forth, you hit the limit sooner than you'd expect.

Why the model "forgets"

When a conversation exceeds the context window, the model doesn't "forget" anything. It simply can't see the older messages, because they are no longer part of the request's input.

Most services handle this in one of two ways.

Truncation: the oldest messages are dropped from the context. The model keeps responding, but it no longer has access to the start of the conversation. Most chat interfaces work this way by default.

Summarization: the first N messages are replaced in the context with a short summary that the model generates itself. Details get lost, but the overall thread survives.

There's also a third approach: RAG (Retrieval-Augmented Generation). Important information lives in an external database and is pulled into the context only when needed. More sophisticated AI systems and enterprise solutions work this way.

Lost in the middle: the hidden problem

Overflow isn't the only issue. Answer quality degrades even before the context runs out.

In 2023, researchers from Stanford and Berkeley described the "lost in the middle" phenomenon: models make much better use of information at the beginning and end of the context. Data that lands in the middle gets processed less effectively, as if the model "doesn't notice" it.

"Language models tend to make worse use of relevant information when it sits in the middle of a long context," according to the

Lost in the Middle study, 2023.

The practical takeaway: put key instructions at the beginning or end of your prompt rather than burying them in the middle of a long document.

How to work with the limit

A few practical strategies:

• Break up tasks — instead of one giant session, split the work into sessions with clear intermediate results
• Put the important stuff first — the system prompt and key constraints work best at the start of the context
• Use models with larger windows — for large documents, pick Gemini 1.5 Pro (2M tokens) or Claude with 200K
• Summarize on your own — before starting a new session, ask the model to sum up the previous one and save that text
• Estimate length up front — 1 page of text ≈ 500 tokens, 1 code file ≈ 1 000–5 000 tokens

What it means

The context window isn't a technical footnote. It's a central parameter in any work with language models. Once you understand the limit, you stop blaming a model's "strange" behavior on a glitch and start structuring your work sessions properly.

The race for bigger context continues, with providers competing to raise their limits. But the engineering question of "what can the model see right now" will stay central for a long time.