RAG evaluation metrics explained with worked examples

What this guide adds

The guide on how to evaluate a RAG system covers the overall process: building a test set, scoring stages, reading failures and catching regressions. This one looks closely at four named metrics and works through the arithmetic, so that when a dashboard reports 0.5 you can say what happened to produce it.

The names and formulas here follow the Ragas documentation. Ragas is an evaluation library, and its authors set out their approach in a paper. Other tools may use the same names with different definitions, so check before comparing numbers across them.

One running example

All the numbers below come from a single hypothetical case, an invented shop's returns policy. The scores are calculated for teaching and are not measurements of any real system.

Question: "How long do I have to return an unopened item, and who pays the postage?"

Reference answer: "Unopened items can be returned within 30 days. The customer pays return postage, unless the item is faulty."

Retrieved chunks, in ranked order:

1. Standard delivery takes three to five working days. (Not relevant)
2. Unopened items may be returned within 30 days of delivery. (Relevant)
3. Gift cards cannot be exchanged for cash. (Not relevant)
4. Customers pay return postage unless the item is faulty. (Relevant)

Generated answer: "You have 30 days to return an unopened item. You pay the return postage, and your refund will arrive within five working days."

Faithfulness

Definition. The share of claims in the answer that the retrieved context supports.

faithfulness = supported claims / total claims in the answer

Split the answer into claims and check each against the chunks:

• Unopened items can be returned within 30 days: supported by chunk 2.
• The customer pays return postage: supported by chunk 4.
• The refund arrives within five working days: not supported. Chunk 1 mentions five working days, but for delivery.

The score is 2 / 3, or about 0.67. Notice how the unsupported claim borrows a phrase from an unrelated chunk, which is a kind of slip a quick reading can miss.

What it misses. Faithfulness measures agreement with the context, not with reality. If chunk 2 were an out-of-date policy saying 14 days, an answer repeating "14 days" would score perfectly and still mislead the customer. It also says nothing about completeness: this answer leaves out the exception for faulty items, and no claim is penalised for that.

Context precision

Definition. How well the retriever places relevant chunks towards the top of its ranking. At each rank holding a relevant chunk, take the precision up to that rank, sum these values, then divide by the number of relevant chunks in the top K.

With K = 4 and relevant chunks at ranks 2 and 4:

• At rank 2, one of the first two chunks is relevant: 1 / 2 = 0.50.
• At rank 4, two of the first four are relevant: 2 / 4 = 0.50.

The score is (0.50 + 0.50) / 2 = 0.50.

Now suppose a reranker moves the same four chunks so that the relevant ones sit at ranks 1 and 3. The values become 1 / 1 and 2 / 3, giving (1.00 + 0.67) / 2, or about 0.83. If they reach ranks 1 and 2, the score is (1 / 1 + 2 / 2) / 2 = 1.00. The set of chunks has not changed, only their order, so this metric rewards ranking.

# Illustrative only: context precision from relevance flags in rank order.
def context_precision(flags):
    relevant_seen = 0
    total = 0.0
    for rank, is_relevant in enumerate(flags, start=1):
        if is_relevant:
            relevant_seen += 1
            total += relevant_seen / rank
    return total / relevant_seen if relevant_seen else 0.0

print(context_precision([False, True, False, True]))  # 0.5
print(context_precision([True, False, True, False]))  # about 0.83

This sample is illustrative and takes the relevance flags as given. Producing those flags is the hard part, and in practice they come from human labels or an LLM judgement. Returning 0.0 when nothing is relevant is a choice made here.

What it misses. It only looks at what was retrieved. Shorten the list to the top two chunks, and the single relevant chunk at rank 2 gives (1 / 2) / 1 = 0.50, exactly the same score as before, even though the postage information is now gone. Spotting that loss is the job of the next metric.

Context recall

Definition. The share of claims in the reference answer that the retrieved context can support. It always requires a reference to compare against.

context recall = reference claims supported by the context / total reference claims

The reference contains three claims: the 30-day window, that the customer pays postage, and the exception for faulty items. Chunks 2 and 4 cover all of them when all four chunks are passed on, so the score is 3 / 3 = 1.00.

Keep only the top two chunks and chunk 4 is lost. Only the 30-day claim is now supported, so the score falls to 1 / 3, or about 0.33. Precision held steady at 0.50 while recall collapsed, which is why the two should be read together.

What it misses. The score is only as good as the reference. A reference that omits an important condition cannot reveal that the context omits it too. This claim-based form also differs from the ID-based recall at k described in the evaluation guide, which counts labelled passages rather than claims, so the two are not interchangeable. If re-chunking keeps breaking your ID labels, a claim-based check can survive it, though it adds the cost and error of an LLM judgement.

Answer relevance

Definition. How directly the response addresses the question, whether or not it is correct. Ragas calls this answer relevancy and calculates it indirectly. An LLM generates several questions that the answer appears to respond to, each is embedded, and the score is the mean cosine similarity between those questions and the user's original question.

For our answer, suppose the generated questions have similarities of 0.91, 0.88 and 0.94 with the original question. These values are invented for illustration. The score is (0.91 + 0.88 + 0.94) / 3 = 0.91.

Compare an evasive response: "Our returns policy was recently updated and can be found in your account area." The questions it implies are about where to find the policy. With invented similarities of 0.62, 0.58 and 0.66, the score drops to 0.62, correctly signalling that the customer has not been answered.

What it misses. It does not test truth. "You have 90 days, and we pay the postage" addresses the question closely, so it could score highly despite being wrong, and faithfulness is what would catch it. Because cosine similarity is technically able to go below zero, the score is not strictly bounded between 0 and 1, and its absolute level depends on the embedding model. Compare runs that use the same one.

Reading the four together

Patterns across metrics tell you more than any single value. The evaluation guide maps observed failures to causes; this table maps score patterns to where to look first.

Practical cautions

• LLM grading adds noise. Splitting claims and judging support are model decisions. Different splits change the denominator, so hand-score a sample and compare before trusting a trend.
• Decide how refusals score. "I could not find that" contains no factual claims to count. Check how your tool treats it, and measure refusal behaviour separately on questions that cannot be answered.
• Avoid universal thresholds. A sound target depends on your content and on the cost of errors. Compare a score with your own baseline, and do not combine the four into one number, as this hides which stage failed.
• Keep run details. Metric definitions can change between library versions, so record the version and models alongside each score.

If you want these measures wired into a pipeline, or need to find out where an existing one loses quality, see RAG development. For background on the system being measured, start with what RAG is and when to use it.

Frequently asked questions

What is the difference between context precision and context recall?

Context precision looks at how well relevant chunks are ranked among those retrieved. Context recall looks at whether the retrieved context covers what the reference answer needs. A system can rank its few results well and still miss necessary information.

Does a high faithfulness score mean an answer is correct?

No. It means the answer's claims agree with the retrieved context. If that context is out of date or wrong, a faithful answer will be wrong too, so correctness needs a separate comparison with a trusted reference.

Which RAG metrics need a reference answer?

Context recall always requires a reference. In the forms described here, faithfulness and answer relevance compare the answer with the context and the question instead. Context precision needs a relevance judgement for each chunk, which may come from labels or a model.

What is a good faithfulness or context recall score?

There is no universal threshold, because acceptable risk varies by use and scores vary by tool and model. Establish a baseline on your own test set, then watch for changes, especially on critical question groups.