RAG Evaluation: Metrics, Golden Datasets & Regression Tests

A RAG system is not well evaluated because its answers sound plausible in a demo. RAG evaluation starts by asking which layer failed: retrieval, context assembly or generation. A metric becomes useful only after that question.

A wrong answer is a symptom, not a root cause. Was the current document never retrieved? Was the decisive passage excluded from the model context? Or did the model misuse evidence that was already available? For the full pipeline, see Build a RAG system the right way. If a PoC already exists and its quality needs to become dependable, AI Production Readiness is the direct next step.

RAG evaluation in one sentence

A production RAG system should be able to assign every wrong or uncertain answer to a testable failure class. A single “answer quality” score can reveal that something is wrong. It cannot show which change actually improves or degrades retrieval, context assembly or generation.

A wrong value is not yet a diagnosis

An employee asks, “What notice period applies to my role?” The system answers, “three months to the end of a quarter.” The current HR policy says “two months to the end of a month”; an older policy contains the value from the answer.

Without diagnosis, changing the prompt or model is an understandable but premature reaction. At least four different causes are possible:

1. Retrieval never found the current policy and returned only the older version.
2. Both documents were found, but chunking, context limits or selection excluded the current passage.
3. The current passage was in context, yet the model preferred the older wording, summarised it incorrectly or answered without reliable support.
4. The expected reference in the test case is itself stale, its policy scope was interpreted incorrectly, or the employee’s role has an exception.

The same visible answer therefore calls for four different checks. That is why metrics should not determine the narrative of an evaluation.

Diagnose a wrong RAG answer

1. 01

   Retrieval

   Was current evidence, permitted for this role, retrieved?

   Inspect recall, filters and index

2. 02

   Context assembly

   Did the decisive passage enter the model context completely and without distracting stale content?

   Inspect coverage, versions and chunk selection

3. 03

   Generation

   Did the model use the supplied evidence correctly and cite it accurately?

   Check correctness, groundedness and citations

4. 04

   Validate the expectation

   Is the reference answer valid for the role, date and policy version?

   Check ground truth and scope

Diagnostic path: retrieval, context assembly, generation

1. Retrieval: was the right evidence found?

This is not about polished answer text yet. Check whether the current policy appears in the retrieved set and whether version, role and permission filters apply correctly. If it is missing, investigate query processing, indexing, embeddings, hybrid search, metadata or filters.

Recall@k is appropriate when the expected evidence must appear in the top-k results. Precision@k helps where irrelevant or stale material burdens the next selection step. MRR or nDCG are useful only when ranking order matters: for example, when the first usable passage has to appear early or several pieces of evidence must be ordered meaningfully.

Search strategy is not a matter of belief. Exact policy names, versions and role identifiers may need lexical search; semantically phrased questions often benefit from vector search. Hybrid Search vs Vector Search for RAG explains why both need to be tested against real cases.

2. Context assembly: did the passage become usable?

A retrieved result is not automatically usable context. The current policy may appear in the top-k results while the actual context window contains only the old passage. Common causes include poor chunk boundaries, reranking, small context budgets, duplicate content or missing version logic.

This is where evidence coverage and the quality of the actual model context matter: is the decisive passage present, complete, current and authorised for this user? Also measure or inspect the balance of relevant and irrelevant evidence, plus stale or duplicate content. More context is not automatically better; irrelevant chunks can displace the claim that matters.

If preprocessing creates an unusable evidence unit, retrieval cannot surface it cleanly. That is why RAG chunking strategies for enterprise data belongs in the same evaluation loop.

3. Generation: was the evidence used correctly?

Even with the right passage in context, the answer must not be assumed correct. Check correctness against business rules, expected values or a reasoned reference answer. Groundedness asks whether the response is supported by the supplied context. Citation accuracy asks whether a cited source actually supports its claim. Instruction adherence and task success matter when the answer must, for example, explain an exception, escalate safely or explicitly decline to invent an answer.

LLM-as-a-judge can scale semantic assessment, but it does not replace a verifiable expectation. Deterministic checks for values, dates, IDs or formats and human review of critical cases remain useful. Version the grader prompt, model and rubric so the measurement method does not move together with the system under test.

4. Validate the expectation: is the ground truth valid?

The golden dataset can be wrong too. For policy questions, it must be clear which version was valid on the relevant date, which role it applies to and whether documented exceptions exist. A reference answer without expected evidence or scoring criteria cannot distinguish a system failure from an incorrect expectation.

This fourth check avoids a costly mistake: optimising a system toward a stale or overly broad test reference.

Golden dataset: a controlled test base, not a benchmark collection

A golden dataset is not a static catalogue of academic questions. It is a versioned control base for real user questions and the evidence an answer is allowed to rely on.

A useful case includes:

• the realistic query type and, where relevant, the actual task,
• expected sources, documents or passages as evidence,
• an expected answer or explicit scoring criteria,
• user role, tenant and relevant permissions,
• an effective date, document version and known freshness risks,
• expected behaviour when evidence is missing or contradictory.

The notice-period case therefore tests more than the words “two months”. It tests whether the current policy is found, whether old versions are suppressed, whether the employee may see the information and whether an exception is handled correctly. Negative cases belong here too: a question must not expose unauthorised evidence in retrieval, context or the answer. This connects evaluation directly to RAG permissions and security trimming.

Metrics follow the failure class

A small, explainable measurement set is better than a long list without a decision context:

NVIDIA’s RAG Blueprint documentation lists dimensions including Answer Accuracy, Context Relevancy and Response Groundedness. That fits this diagnostic principle: retrieved context and generated output are not the same quality question. Source: NVIDIA RAG Blueprint – Evaluate.

Regression, not demo scoring

Once the golden dataset covers real failure classes, it becomes a regression suite. Every change to chunking, retrieval, reranking, prompting, the model or the data should run against the same controlled base. The same applies to embeddings, query rewriting, hybrid weighting, context size, source connectors and permission logic.

The question is not whether a new version “feels better”. Record which cases improve, which regress, and how cost or latency shifts. A global average must not conceal a critical regression in policy, pricing, permissions or compliance cases.

Production monitoring and drift

Regression testing protects known cases before rollout. Production introduces new documents, changed language, new roles, altered permissions and unexpected questions. Monitoring therefore complements the golden dataset with signals such as retrieval failures, empty or overloaded contexts, citation errors, escalations, no-answer rates, latency and cost.

Production signals are not automatic proof of a model failure. They provide candidates for new or refined golden-set cases. This keeps the dataset controlled and lets it grow only through reviewed real failures.

Architecture view: make failures locatable

A production RAG system should make failures locatable. If retrieval, context assembly and generation are assessed only through one “answer quality” score, it remains unclear which change actually improves or degrades the system.

The architecture therefore needs separately inspectable artefacts: query and user context, retrieved results with version and permission data, the chunks actually passed to the model, the answer and its source mapping, and the version of the evaluation rule. Every change to chunking, retrieval, reranking, prompting or the model should be regression-tested against the same golden dataset.

This is not extra bureaucracy. It is what lets a wrong answer lead to a concrete technical decision instead of optimisation at the wrong layer.

Next step

If an existing RAG system produces plausible answers but quality, permissions or regressions cannot be diagnosed reliably, another prompt-tuning cycle is usually not the right next move.

I help teams evaluate existing RAG and LLM systems using realistic test sets, failure analysis and explicit go-live criteria: AI system evaluation and production readiness.

For a new build or technical integration into existing systems, see RAG implementation & AI integration.