RAG evaluation is two problems, and most teams measure one

A retrieval-augmented system gave a wrong answer. Was that because the right document wasn't retrieved, or because the model had the right document and answered badly?

Those have completely different fixes — one is an index and chunking problem, the other is a prompting and model problem — and in the final output they look identical.

If your evaluation reports one number, you cannot tell them apart, and you will spend money on the wrong half.

Measure the two stages separately

Retrieval. For each question, was the passage containing the answer in the retrieved set? This needs a labelled set: question, and which document actually contains the answer. Building that is tedious and it's the highest-leverage thing you can do.

Two numbers worth tracking: how often the answer was retrievable at all, and where in the ranking it landed. A system that finds the right chunk at position nineteen when you pass five to the model has a ranking problem, not a coverage problem, and those get fixed differently.

Generation, given retrieval. Hold the retrieved context fixed — ideally hand it the perfect context — and ask whether the answer is correct. This isolates the model from the index.

The two numbers together give you a diagnosis the combined score never will:

The distraction failure

That third row is the one people miss, and it's specific to RAG.

Retrieval works. Generation on clean context works. The combined system underperforms both — because retrieval returned the right chunk plus four plausible, irrelevant ones, and the model used the wrong one.

More retrieved context is not monotonically better. Passing ten chunks instead of three can lower accuracy by giving the model more attractive wrong answers. This shows up nowhere in retrieval metrics, because retrieval succeeded, and nowhere in isolated generation metrics, because you gave it clean context.

Test it directly: run the same questions with clean context and with realistically noisy context. The gap is your distraction cost, and it's often larger than the model upgrade you were considering.

Grounding is the criterion that matters most

The failure that hurts in production is not "wrong." It's confidently wrong with a citation that doesn't support it.

Three checks worth making explicit criteria:

• Is every factual claim traceable to a retrieved passage? Not "does it cite" — does the cited passage actually contain the claim.
• Are the citations real? Models produce plausible document IDs and page numbers that don't exist. A reviewer under time pressure will accept them.
• Does it say when it doesn't know? A system that answers everything has no abstention behaviour, and its confidence carries no information.

That last one deserves its own metric. Measure the abstention rate and the accuracy conditional on answering. A system that answers 60% of questions at 95% accuracy is often more valuable than one that answers 100% at 80% — and the single combined number ranks them the other way round.

Freshness is a failure mode, not a nice-to-have

Your index has a state. Documents get superseded. Policies get updated. If the retrieval layer serves last quarter's version, the model will answer correctly about last quarter, confidently, with a real citation.

This is nearly invisible in evaluation, because most eval corpora are static snapshots where nothing was ever superseded.

Put it in the test set deliberately: include documents with superseded versions present in the index, where the correct behaviour is to use the current one. In regulated workflows this is frequently the highest-consequence failure available and it's the one nobody tests.

What to build first

If you have nothing, build in this order:

1. A labelled retrieval set. Fifty questions with the document that answers each. Tedious, unglamorous, unlocks everything else.
2. A clean-context generation set. The same questions with perfect context handed in.
3. The combined run. Now the difference between it and step 2 is your retrieval-plus-distraction cost, quantified.
4. Grounding criteria on the combined run — traceability, citation validity, abstention.
5. A superseded-document tier, once the first four are stable.

Each step answers a question the previous one couldn't, and each one points at a specific team.

The one-line version

A RAG system with one number is a system where you can see something is wrong and cannot see which half.

Two numbers cost slightly more to produce and turn a symptom into a diagnosis.