Evaluating RAG Systems: Faithfulness, Answer Relevance, Context Precision

The first useful thing to say about evaluating a retrieval-augmented system is that you are not evaluating a model. You are evaluating a pipeline. The model is one component in it, and most of the time the model is not the part that is failing. The team that confuses the two ends up swapping the engine when the brakes are the problem.

I keep coming back to a watchmaker’s bench as the right mental picture. A mechanical movement has dozens of moving parts: mainspring, escapement, balance wheel, gear trains. When the watch runs fast, the watchmaker does not throw out the movement and buy a new one. They measure each component against its own calibration, find the one that drifted, and adjust it. RAG evaluation is the same discipline. The metrics that have emerged in the last two years are calibration gauges for specific components of the pipeline, not a single quality score for the watch.

That distinction is what makes the move from model evaluation to RAG evaluation more than a vocabulary change. Model evaluation asks whether the parameters learned the right things. RAG evaluation asks whether the pipeline, prompt to retrieval to generation to citation, holds end to end on your traffic. The first is a question for the model lab. The second is a question for whoever ships the system.

The three metrics the field has converged on

If you read the documentation for RAGAS, TruLens, and a half-dozen smaller frameworks side by side, the naming drifts but the shape is the same. Three metrics anchor the discipline, and each one isolates a different failure mode in the pipeline.

Faithfulness asks whether the generated answer is supported by the retrieved context. Every claim in the answer is decomposed, and each claim is checked against the documents that retrieval handed to the generator. If a claim cannot be found in the context, it counts against the score. This is the metric that catches the hallucination that hurts most, the one where retrieval did its job, the documents were on the desk, and the model wrote a confident sentence the documents do not actually support. Es et al. introduced the formalisation in the RAGAS paper; the rest of the field largely adopted it.

Answer relevance asks whether the answer addresses the question the user asked. A faithful answer can still be useless if it answers a different question, or wanders into background context the user did not ask for. The common implementation reverses the question: the framework asks a model to generate plausible questions for the given answer and measures how close those generated questions are to the original. High similarity means the answer is on-target. Low similarity means the answer is grammatically clean but topically adrift.

Context precision asks whether the documents retrieval handed to the generator were actually relevant to the question. It scores the top-K retrieved passages by relevance and rewards rankings that put the relevant ones on top. A retrieval system that returns ten passages where the right one is buried at rank nine is technically correct and operationally broken; the generator’s attention budget is finite and the wrong neighbours crowd out the right answer.

Two more metrics earn their place in production, even when they sit one step off the headline triad.

Context recall measures whether retrieval surfaced everything needed to answer the question, not just whether what it surfaced was on-topic. Recall and precision pull in opposite directions, so the two are read together. High precision with low recall means retrieval is conservative and missing things. High recall with low precision means retrieval is greedy and burying the signal in noise.

Noise sensitivity is the newer one. It measures how the answer changes when irrelevant or contradictory passages are deliberately added to the context. A robust system holds steady. A brittle one gets pulled toward whichever passage was loudest. This is the metric most teams skip until a production incident teaches them why it matters.

How the metrics actually get computed

Most of these metrics are computed by an LLM acting as the judge. The pattern is consistent: a stronger model reads the question, the retrieved context, and the generated answer, then returns a score. Zheng et al. studied the technique in “Judging LLM-as-a-Judge” and found that judge models correlate well with human raters on most tasks, with known biases worth designing around.

The biases matter. Judge models prefer longer answers, tend to favour the first response in a pair, and inherit any positional or stylistic preferences of the underlying model. The mitigations are by now standard practice: use a different model family as the judge than the one being judged, randomise position in pairwise comparisons, decompose claims explicitly rather than asking for a holistic verdict, and validate the judge on a small human-labelled set before trusting it at scale. None of these eliminate bias; they keep it inside known bounds.

The other quiet cost is ground truth. Faithfulness can be computed with no ground-truth answer because the standard is “supported by the retrieved context,” which the context itself provides. Context recall genuinely needs a labelled answer to compare against. Teams that build small, honest, human-labelled evaluation sets from real traffic out-ship teams that try to substitute synthetic data for the judgement call. The work of curating fifty real questions with real correct answers is more valuable than the work of generating five thousand synthetic ones.

Where to measure, in the pipeline

The point of these metrics is not to produce a dashboard. It is to tell you which component to fix when the system regresses. Each metric attaches to a specific stage.

When faithfulness drops, the suspect is the generator or the context assembly; when context precision drops, the suspect is retrieval; when answer relevance drops, the suspect is the prompt or the model’s handling of ambiguous queries. The diagnostic value is in the attribution, not in any single number.

What this looks like in production

The version of this discipline that survives contact with a live system is small and stubborn rather than large and impressive. A few patterns hold up.

Sample a small slice of real traffic, daily, and run the same metrics on it that ran in pre-release evaluation. Set thresholds for each metric per workflow, not a single global threshold; a customer-support assistant tolerates different precision than a legal research tool. When a metric breaches its threshold, page on the metric, then walk the pipeline to find the stage that moved. Keep a held-out evaluation set for regression catching at release time, and a rolling production sample for drift detection between releases.

The connection back into the rest of the wrapper is direct. A faithfulness regression is rarely a model problem; it is almost always a retrieval or document chunking problem upstream. A context-precision drop is a retrieval-tuning question, not a generation question. The metrics earn their keep when they shorten the time between a regression and the fix, and they only do that when the team trusts the attribution enough to act on it.

The broader practice of standing up an evaluation discipline across the whole system, from leaderboard score through harness proof, and of running it continuously on production traffic, is what turns these numbers from a quarterly review artifact into a runtime control. The metrics on this page are the gauges. The discipline of reading them every day is what calibrates the watch.