Enterprise RAG: 80% of the Problems Are in the Data, Not the Model

A RAG prototype can be assembled in a week and shown off in a demo. It looks magical: the model answers from your documents and does not hallucinate from general knowledge. But a couple of weeks into production, the system starts mixing up versions, loses context and confidently produces answers from sources that do not exist. This is the typical path for most enterprise RAG. And the culprit is not the transformer, but the data and the architecture.

Where RAG breaks

When you scale from prototype to production, an unpleasant truth emerges: 70-80% of RAG problems stem from data management, indexing and retrieval, not from the capabilities of the language model itself. However good GPT-4 or Claude may be, if the data is poorly indexed, the system will give wrong answers. This becomes especially visible when a RAG system goes from 100 to 10,000 documents.

Here are the real reasons RAG fails in the enterprise:

• Document version mix-ups: old versions stay in the index, and the system does not know which document is current. The user gets an answer based on a two-year-old policy.
• Lost context: the system does not remember what was discussed earlier in the chat, repeats itself or contradicts itself.
• Poor chunking: documents are split by size rather than by meaning. Logic falls apart between chunks, and the system cannot see the connections.
• No reranking: BM25 pulls in a lot of noise, and the system cannot tell relevant documents from a chance keyword match.
• Low-quality embeddings: the vectors are trained on a general corpus and do not understand your specifics and terminology.
• No feedback loop: nobody tracks wrong answers, so the system does not learn from its mistakes.

How to build RAG properly

At AlpinaGPT we worked backwards: first we gathered requirements for an ideal system, then looked at which specific problems block them. The result is an architecture that has been battle-tested with corporate clients. These are the key components:

• Meaning-based chunking: we split by document structure, headings and semantic blocks, not by size. That way context does not fall apart between chunks.
• Data versioning: each document version is indexed separately with a date tag. The system knows which document is current.
• Two-stage retrieval: first fast BM25 (keywords), then neural reranking (semantics). This is cheaper than searching everything through embeddings.
• Context across messages: the system remembers the entire chat history and does not repeat what has already been explained to the user.
• Feedback: we track wrong answers and retrain the ranker on those examples.
• Separate indexes by type: policies, instructions, FAQs and code are indexed differently.

What this means

The RAG of the future is not just embeddings and vector search. It is document version management, proper semantic chunking, multi-stage retrieval and continuous feedback. If half the answers from your RAG are wrong, the problem almost certainly is not the GPT-4 or Claude model. The problem is how you prepare the data, how you split it, how you index it and how you collect feedback. Rework that whole pipeline and quality jumps. This is what we learned from AlpinaGPT.