RAG still hallucinates. Grounding takes more than retrieval

Retrieval-augmented generation is the standard answer to hallucination: give the model the documents and tell it to answer from them. It helps. It does not finish the job.

The evidence

The clearest test so far comes from law, where a wrong answer has consequences. Stanford's RegLab and HAI evaluated commercial legal research tools built on retrieval. Lexis+ AI and Ask Practical Law AI produced incorrect or misgrounded answers on more than 17 percent of benchmark queries, and Westlaw's AI-Assisted Research on more than 34 percent. The authors found retrieval reduced hallucination compared with general-purpose models, and that it is no panacea. Vendors disputed the definitions; the failure modes the paper documents are still worth designing against.

How RAG fails

• Retrieval misses. The decisive document is not in the top results, so the model answers from the next best thing.
• Chunks lose meaning. A clause split from its definition, a table split from its header, a function split from its modifier.
• Citations that do not support the claim. The model cites a real passage that says something different.
• Lost in the middle. Models attend less reliably to evidence in the middle of a long context.
• Sycophancy. A question with a false premise gets an answer that accepts it.

The proposal: grounding as a pipeline with checks

1. 01Chunk by structure. Split documents along their own structure: sections and clauses for contracts, functions for code, rows with headers for tables. In ContractLens each chunk is a complete function or modifier.
2. 02Retrieve two ways, then rerank. Vector search for meaning, full-text search for exact names and numbers, fused, then reordered by a cross-encoder so the best evidence comes first.
3. 03Cite at passage level. Pass sources to the model as citable blocks, so every claim points to a document, page and passage. In Holmes each claim ties to a document, a page and a literal quote that the user can click through to.
4. 04Verify after generation. Check that each cited passage exists in what the model was shown and actually supports the sentence; drop or flag claims that fail. ContractLens validates every evidence reference against the retrieved sources and clamps line ranges to them.
5. 05Abstain on purpose. When the evidence is not there, the right answer is "the documents do not say". Reward that in evaluation instead of penalising it.
6. 06Challenge the premise. Instruct the model to check the question's assumptions against the sources before answering.
7. 07Evaluate retrieval and answers separately. Measure whether the right passages were retrieved, and separately whether the answer is faithful to them. A single end-to-end score hides which half is broken.

Where memory fits

Retrieval answers from documents as they are. Some questions are about what changed over time: what was agreed, what was superseded. That needs a memory layer that keeps decisions with their history, which we built in DeltaMemory.

What to measure

• Retrieval recall. Whether the passage that holds the answer is in the top results, per question type.
• Faithfulness. The share of answer sentences fully supported by their cited passages.
• Abstention quality. How often the system says it cannot answer when the documents do not contain the answer, and how rarely it says so when they do.
• Premise checks. On questions with a false premise, how often the answer corrects it.

How we would start

Collect fifty real questions with known answers and the documents that contain them, and score your current system on retrieval and faithfulness separately. The weaker half tells you where to work. This is how we start RAG development and AI memory projects, and when retrieval is not the right tool, we say so: see fine-tune, retrieve or prompt.