How to evaluate a RAG system

Why a RAG system needs its own evaluation

A RAG system has several stages that can fail independently. A parser can garble a table, retrieval can miss the right passage, a reranker can push it down the list, and the model can misread it or go beyond it. By the time you see a poor answer, the original cause may be several steps back.

Trying a handful of questions by hand does not reveal much. It tends to favour the questions you already know will work, and it gives you no way to tell whether a change helped overall or just fixed the example in front of you. A repeatable evaluation replaces that impression with evidence you can compare between versions.

Step 1: build a test set from real questions

The test set is the most valuable part of the evaluation, and it is worth more effort than the choice of scoring tool. Each item should contain a question, the passages that answer it and, where practical, a reference answer or a list of facts the answer must include.

Real user questions are the best source when they are available, from search logs, support tickets or a pilot. If none exist yet, ask people who know the content to write questions in the way users would. Generating questions with an LLM can help to widen coverage, but such questions often echo the wording of the source passage, which makes retrieval look better than it will be in use. Review them and do not rely on them alone.

Aim to cover different kinds of question rather than many variations of the same one:

• Simple lookups answered by a single passage.
• Exact-term questions that mention a product code, error message or policy name.
• Questions spanning several passages, where the answer has to be assembled.
• Differently worded questions, where users' terms do not match the documents.
• Unanswerable questions, where the right behaviour is to say the content does not cover it.
• Follow-ups in a conversation, if your product is conversational, since they depend on earlier turns.
• Access-restricted questions, if permissions apply, to confirm that one user cannot draw on another's documents.

Label relevance by a stable identifier where you can, but remember that re-chunking changes chunk boundaries and may invalidate chunk IDs. Labelling against a document and section, or against the required text itself, lets the test set survive changes to your chunking strategy.

Step 2: measure retrieval on its own

Retrieval is usually the first place to look, because the model cannot use a passage it never receives. It is also cheap to test, since no generation is needed. Common measures include:

• Recall at k. Of the relevant passages, what share appeared in the top k results? This matters most when you pass the top k to the model, because a passage outside the cut-off might as well not exist.
• Precision at k. What share of the top k results are relevant? Low precision wastes context and gives the model more distracting material.
• Mean reciprocal rank. How high does the first relevant result appear, averaged across questions? This suits questions that one passage can answer.
• nDCG. A rank-weighted measure that also supports graded relevance, such as "fully answers" versus "partly useful".

The first and third are simple enough to write yourself:

# Illustrative only. Assumes results are ranked IDs
# and relevant is a non-empty set of IDs.
def recall_at_k(results, relevant, k):
    return len(set(results[:k]) & relevant) / len(relevant)

def reciprocal_rank(results, relevant):
    for rank, item in enumerate(results, start=1):
        if item in relevant:
            return 1 / rank
    return 0.0

Score each pipeline stage where you can. If a passage is present in the first-stage candidates but missing after reranking, the problem lies in the reranker, not the index. If it is absent from the candidates altogether, look at parsing, chunking, the query or the search method. The guide on hybrid search and reranking covers the fixes that most often follow from this kind of finding.

Step 3: measure answer quality

A good retrieval score does not guarantee a good answer. Answer quality is better treated as several questions than as a single score:

• Faithfulness. Is each claim in the answer supported by the supplied passages?
• Correctness. Does the answer match the reference? An answer can be faithful to a retrieved passage and still be wrong if that passage is out of date.
• Relevance. Does it address what was asked, instead of offering true but unrelated information?
• Completeness. Does it include all the required facts, particularly on multi-passage questions?
• Citation accuracy. Does each cited source really support the sentence it is attached to?
• Refusal behaviour. Does it decline on unanswerable questions and avoid declining on answerable ones?

Human review and model-assisted grading

Human review is the most trustworthy method for most teams, but it is slow, and reviewers can disagree. Written criteria with examples of each score make their judgements more consistent, and having two people score a sample shows how clear the criteria really are.

Model-assisted grading uses an LLM to compare an answer against the passages or a reference. It allows frequent runs, but the grader is itself imperfect. Before trusting its numbers, have people score a sample, run the grader on the same items and read the disagreements. Asking for one narrow judgement at a time, such as "is this sentence supported by this passage?", is generally easier to verify than asking for an overall quality rating.

Simple code checks also have a place. Whether a required figure appears, whether a citation points to a retrieved passage and whether the output matches a required format can all be tested without a model at all.

Step 4: read the failures, not just the averages

An average can hold steady while one important question type collapses. Break results down by question type, source and any segment that matters to you, then read a sample of failures in full. For each one, trace the run and record where it first went wrong:

Keeping a tally of these causes is often the most useful output of the whole exercise, because it shows which fix is likely to help the most questions.

Step 5: compare changes and catch regressions

Treat the evaluation like a test suite. Change one thing at a time, such as the chunk size or the reranker, and compare the new run with the previous one question by question. Record the configuration alongside the scores, including the prompt, the models and the index version, so that results can be reproduced later.

LLM output can vary between runs, so a small movement in a score may be noise. Decide in advance which results would block a release: a drop on a critical question group may matter more than a small gain in the overall average.

Include latency and cost in the comparison as well. A change that improves quality slightly but makes every answer much slower may not be worth having, and only you can weigh that for your product.

Evaluating after launch

An offline test set only covers what you thought of. Once people are using the system, it helps to log questions, retrieved passages and answers, subject to your privacy obligations. Thumbs-up and thumbs-down ratings are useful, although they are sparse and tend to reflect strong reactions. Follow-up rephrasing, source clicks and escalation to a person can provide further signals, but each needs careful interpretation.

Questions that fail in use should be reviewed, labelled and added to the test set, so that it steadily comes to reflect real usage. Content changes matter here too: an index update can alter results without any code changing, so run the evaluation after significant content updates as well.

Common evaluation mistakes

• Testing only answerable questions. A system that always answers will look fine until someone asks about something it does not cover.
• Tuning against the test set until it stops being a test. Keep a held-out group of questions that you do not inspect while tuning.
• Trusting one combined score. It hides which stage failed, and so which fix to try.

If you need help setting up this kind of measurement or tracing where an existing pipeline loses quality, see RAG development. Retrieved text can also be hostile, so a security-minded test set should draw on the guide to LLM guardrails and prompt injection defence.

Frequently asked questions

What is the difference between retrieval and answer evaluation?

Retrieval evaluation asks whether the right passages were found and ranked well. Answer evaluation asks whether the generated response is correct, supported by those passages and useful. Scoring them separately shows which part needs fixing.

How many questions does a RAG test set need?

There is no universal number. Start with a set small enough to review by hand but broad enough to cover your main question types, then add to it as real failures appear. Be cautious about small score differences on a small set.

Can an LLM grade RAG answers reliably?

It can help, especially with faithfulness and relevance checks at a scale people cannot review, but it makes mistakes and its behaviour can change with the model or prompt. Compare its judgements against human labels and spot-check them regularly.

What should a RAG system do when the answer is not in the content?

It should say that it could not find an answer rather than guess. Include unanswerable questions in your test set so that this behaviour is measured directly and does not get worse unnoticed.