Evidence note: Models Desk implementation guide as of 2026-08-18. Metrics and thresholds here are starter practices, not universal benchmarks. Validate them against your corpus, user risk, and production requirements.
Quick answer
A useful RAG evaluation workflow starts with a frozen question set. For every question, define the expected source, inspect whether retrieval finds that source, judge whether the answer stays faithful to the retrieved evidence, and record failures by layer. Do not tune prompts blindly. If the right document is absent from top-k retrieval, the problem is usually parsing, chunking, metadata, embeddings, or filters before it is a generation problem.
Why RAG evaluation feels confusing
RAG systems fail in layers. The final answer may look wrong, but the root cause may be a malformed PDF, a chunk split through the middle of a policy, missing metadata, an embedding mismatch, an over-broad filter, or a prompt that allows unsupported claims. One score cannot explain all of that.
The practical answer is not to build a giant evaluation platform on day one. Build a small, disciplined review loop:
- Questions that represent real user work.
- Expected source documents for each question.
- Top-k retrieval inspection.
- Answer faithfulness review.
- A failure log that names the failed layer.
If you have not built the ingestion path yet, start with Build a RAG App: From Documents to Retrieval. This article assumes documents are already parsed, chunked, embedded, and searchable.
Step 1: create a frozen question set
Start with 20 to 50 questions. That is enough to catch obvious regressions without slowing the team. Each row should include:
question: written in user language, not developer language.expected_source: document or URL that contains the answer.expected_section: heading, anchor, or chunk range when available.answer_type: fact, policy, procedure, comparison, refusal, or freshness check.risk_level: low, medium, high.
Include negative questions too. A RAG app that always answers is not grounded; it is just confident. Add questions where the correct response is “the provided sources do not say.”
Step 2: measure retrieval before answer quality
Retrieval recall is the first gate. If the expected source is not in the retrieved context, a faithful answer is unlikely. A simple starter metric is recall@5: did the expected source appear in the top five retrieved chunks or documents?
| Metric | Question it answers | Starter target |
|---|---|---|
| recall@3 | Does the correct source appear very early? | Useful for precise docs |
| recall@5 | Does retrieval find the right evidence at all? | Good first gate |
| no-hit rate | How often does search miss completely? | Track weekly |
| stale-hit count | Are old sources still being retrieved? | Should trend to zero |
| p95 retrieval latency | Is search fast enough for the product? | Depends on UX |
Step 3: judge faithfulness with evidence visible
Faithfulness means the answer is supported by the retrieved context. It does not mean the answer is eloquent. During review, show the evaluator the question, the answer, and the retrieved chunks. Ask three questions:
- Does the answer make a claim that is absent from the retrieved evidence?
- Does the answer cite the correct source for the claim?
- Does the answer refuse when evidence is insufficient?
Use a simple label set: faithful, partially faithful, unsupported, refusal should have happened. This is usually clearer than a vague one-to-five score.
Step 4: keep a failure log
A failure log prevents review meetings from collapsing into “the model is bad.” Every failed row should name the layer that probably caused the issue.
| Category | What it means | Next action |
|---|---|---|
| Parse failure | Expected text never entered the clean corpus | Fix parser or OCR path |
| Chunk failure | Answer span was split or stripped of heading context | Adjust split strategy |
| Metadata failure | Filters excluded the right source or grouped it badly | Fix source metadata |
| Embedding failure | Semantic similarity misses domain language | Try hybrid search or different embedding model |
| Prompt failure | Answer makes unsupported claims despite good evidence | Tighten citation and refusal rules |
| Freshness failure | Old content remains in the index | Delete-and-reindex by source ID |
Step 5: build a small dashboard
Your first dashboard does not need to be fancy. It should separate retrieval health from generation quality. Put these numbers on the same page:
- recall@5 on frozen set
- faithful answer rate
- no-hit rate
- stale source hits
- p95 retrieval latency
- open failures by layer
Review cadence
For a small team, run this loop weekly while the corpus and prompts are changing. After launch, keep a daily automated smoke test and review failures when documents change. Do not let product demos become the only evaluation method.
Final checklist
- Frozen question set exists and is versioned.
- Each question has an expected source.
- Top-k retrieved chunks are logged.
- Faithfulness review sees evidence, not just final answers.
- Failures are categorized by layer.
- Prompt changes require evaluation results before shipping.
RAG evaluation is not about proving the system is perfect. It is about making failures specific enough that engineers can fix them.