Claude Code helped push a legal RAG pipeline to 0.791, but the ARLC 2026 final hit a scaling wall

The ARLC 2026 case shows how fragile RAG can be on real-world tasks. Over five days, the author and Claude Code raised a legal pipeline's warmup score from 0.034 to 0.791, then ran into a hard scaling wall in the final.

From a bug to a breakthrough

The challenge required more than answering questions about court rulings and laws: it demanded exact source page citations. That made grounding a multiplier on the overall score, so even strong answers with weak references nearly zeroed it out. That is what happened at the start: the first version scored 0.034, even though answer accuracy was already high on some questions.

The problem lay not in the model or in retrieval, but in the output format. The author spent three attempts before spotting a simple error: the doc_id field was sent as the file name including .pdf, while the system expected an identifier without the extension. That one fix raised grounding from 0.05 to 0.55 and the overall score from 0.034 to 0.438. The pipeline then reached 0.791 on warmup over 17 iterations.

The F-beta math with β=2.5 also helped. It showed that extra pages hurt more than they seem to, and each superfluous reference can cost 10–22% of grounding quality.

Architecture and techniques

The best result came from a pipeline that indexed whole PDF pages rather than chunks. That is an important choice for legal RAG: if the metric checks for a hit on a specific page, chunking complicates mapping back to the source and adds noise. Search used a hybrid of BM25 and embeddings merged via RRF, with OCR added for scans. On top of that, the author capped the number of pages returned and routed comparison questions, which require matching two documents, separately.

• Page-level retrieval instead of chunks
• BM25 + embeddings + Reciprocal Rank Fusion
• OCR fallback for empty or scanned pages
• Cap on pages per answer, depending on question type
• Fast deterministic branches for simple cases

"Validate the output format first.

Improve quality afterward."

Claude Code played a separate role in the case. With it, the author wrote about 3000 lines of code across seven modules in five days and shipped 17 versions instead of the 3–5 they estimate manual work would have allowed. The agent sped up edits, refactoring, submission runs and diff checks before sending. But the strategic decisions stayed with the human: which metrics to fix first, how to interpret regressions, and when not to touch an already tuned prompt.

Where it broke

The warmup corpus held 30 documents and 100 questions; the final had 303 documents, 4244 pages and 900 questions. That is where it became clear that a pipeline that behaves well on a small set is not guaranteed to carry over to a large one.

First, a cache bug surfaced: the system mistakenly indexed the 30 warmup documents instead of the 303 final ones, pushing null answers up to 37. After the cache was cleared the problem went away, but the main failure remained: the final score fell 42%, to 0.457.

The root causes were architectural. The huge DIFC Courts Rules document began polluting results for many legal queries, consultation papers with identical numbers but different years broke disambiguation, and the regex for law numbers replaced answers about fines with law numbers. A quick batch of eight fixes looked reasonable but worsened the balance of metrics overall: some deterministic accuracy improved, while grounding and the total score fell even further.

This write-up is valuable because it does not sell AI-assistant magic. Claude Code delivered speed, but it did not remove the core engineering work: validating the format, computing the metric, testing one change at a time, and checking the system at a scale close to production. The author's main conclusion is blunt: if the eval set is several times smaller than the production corpus, you are testing luck, not retrieval.

What this means

For teams building RAG products, this is a useful cold shower. The winner is not the most complex stack but the most disciplined one: exact output format, a clear metric, minimal noise in references, and testing at real scale. AI coding assistants already deliver a serious speedup, but they do not yet replace engineering judgment or responsibility for architectural decisions.