Evaluating RAG Systems: Metrics for Retrieval and Answer Quality

Every stage of a RAG pipeline can be built correctly and the system can still give a wrong or unhelpful answer. Once documents are uploaded, chunked, embedded, retrieved, and assembled into a prompt, one question remains unanswered: how do you know if any of it actually works?

Most teams find out from user complaints. That is the slowest and most expensive feedback loop available. A better approach is to evaluate the system deliberately, at two separate layers, before users ever notice a problem.

Two Layers, Not One Score

A RAG system fails in one of two places:

retrieval finds the wrong evidence, or not enough of it
generation produces a bad answer even from good evidence

Treating "the answer was wrong" as a single verdict hides which layer actually broke. Evaluation needs to score retrieval and generation separately, then combine them.

Question
  -> retrieval quality (did we find the right chunks?)
  -> generation quality (did the model use them correctly?)
  -> end-to-end quality (was the user actually helped?)

Retrieval Metrics

Retrieval evaluation compares what was returned against what should have been returned. It requires a labeled set of questions with known correct source chunks or documents.

Common metrics:

Recall@k — of the relevant chunks that exist, what fraction appear in the top-k results?
Precision@k — of the chunks returned, what fraction are actually relevant?
MRR (Mean Reciprocal Rank) — how high up the ranking is the first relevant result?
nDCG — rewards relevant results more when they rank higher, not just when they appear at all

These metrics do not require an LLM at all. They are fast, deterministic, and cheap to run on every change to chunking, embeddings, or reranking.

Building a Golden Test Set

Retrieval metrics are only as good as the labeled questions behind them. A practical way to build one:

pull 50-200 real or realistic user questions
for each, identify the document and ideally the exact chunk that should answer it
store this as a fixed evaluation set, separate from production data
re-run it after any pipeline change

This set becomes the regression suite for the retrieval layer. Without it, every change to chunk size, embedding model, or reranker is a guess.

Generation Metrics

Even with perfect retrieval, the model can still misuse the context it was given. Generation evaluation checks the relationship between the answer and the retrieved chunks, not the answer in isolation.

Faithfulness / groundedness — is every claim in the answer supported by the retrieved context?
Answer relevance — does the answer actually address the question asked?
Context precision — did the model use the relevant chunks, or lean on noise?
Citation accuracy — do the cited sources actually contain the claim attributed to them?

These checks catch hallucination even when retrieval did its job correctly.

Using an LLM as a Judge

Faithfulness and relevance are hard to score with simple string matching. A common pattern is to use a second LLM call as a judge:

Given: question, retrieved context, generated answer
Ask the judge model:
  - is every factual claim supported by the context? (yes/no + explanation)
  - does the answer address the question? (score 1-5)
  - are citations accurate? (yes/no)

LLM-as-judge is not perfect, but it is far more scalable than manual review, and it catches classes of errors that automated string metrics miss entirely. Treat judge scores as a strong signal, not ground truth — spot-check its verdicts against human review periodically.

End-to-End and Human Evaluation

Automated metrics predict quality; they do not guarantee it. A smaller layer of human evaluation should sit on top:

weekly manual review of a random sample of real conversations
thumbs up / down feedback captured directly in the chat UI
flagged conversations routed to a review queue

Human review is slower and more expensive per sample, so it should target ambiguity that automated metrics cannot resolve, not replace them.

What to Evaluate, and When

Stage	Metric type	When to run it
Retrieval	Recall@k, MRR, nDCG	on every pipeline change (CI)
Generation	Faithfulness, relevance	on every prompt or model change
End-to-end	Human review, feedback	continuously, sampled

Running retrieval and generation metrics in CI, against the golden test set, turns "did this change help or hurt?" into a number instead of a guess.

Common Evaluation Mistakes

evaluating only final answers, never retrieval in isolation
no golden test set, so every regression is invisible until a user hits it
treating LLM-judge scores as exact and final rather than directional
evaluating once at launch and never again as content and usage evolve

A RAG system is not static. Documents change, user questions drift, and models get swapped. Evaluation has to run continuously, not once.

Final Takeaway

Retrieval metrics and generation metrics answer different questions, and both are needed to know whether a RAG system is actually working. A golden test set turns quality from a feeling into a measurable, repeatable check that can run before every change ships.

Knowing a system's quality is only useful if you can act on it in production. The next article covers observability: how to log, trace, and monitor a RAG system so failures are visible in real time, not discovered after the fact.

Previous: From Retrieved Chunks to Prompt Context and Chat Response

Continue with: Observability for RAG: Logging, Tracing, and Catching Failures in Production(scheduled)