Evaluate Retrieval: Recall, Faithfulness, and RAG QA

Quick answer

RAG evaluation for recall and faithfulness should separate retrieval from generation before any model debate. First ask whether the right evidence appeared in top-k; then ask whether the answer stayed faithful to that evidence. Use labeled question-passage pairs from real user queries, track recall@k plus MRR/nDCG, and calibrate any LLM-as-judge against human labels. Re-run eval when embedders, chunkers, ACL filters, rerankers, or generators change. Otherwise teams optimize fluent hallucinations instead of fixing missing evidence (how to read AI leaderboards).

Key takeaways

• Fix recall@k before blaming the generator—most RAG incidents are retrieval misses.
• Faithfulness metrics catch answers that sound correct but invent details beyond chunks.
• Holdout sets must be excluded from prompt tuning and judge calibration leakage.
• Log citation chunk IDs to enable human audit samples without replaying full sessions.
• End-to-end latency belongs in the same scorecard as quality (inference economy).

Prerequisites

• Stable ingestion pipeline producing chunk IDs (see chunking guide).
• 200+ labeled examples to start; expand when metrics stabilize across product areas.
• Document version metadata to detect stale gold labels after corpus updates.
• Baseline retrieval config (embedder, k, filters) frozen while testing one variable at a time.
• Agreement process with legal/privacy if labels come from production logs.

Step 1: Define layers and metrics

Public frameworks such as Ragas package several generation metrics; treat them as accelerators, not substitutes for task-specific gold data.

Step 2: Build a gold question set

Sources of questions

• Redacted production user queries (best distribution match).
• Support ticket subjects and agent escalations.
• SME-authored questions from onboarding and training scenarios.
• Adversarial cases: rare SKUs, negation, multi-hop entity joins (GraphRAG eval extensions).

Annotation guidelines

For each question, annotators mark:

1. Gold chunk IDs (one or more) that contain sufficient evidence.
2. Acceptable alternates if multiple passages answer equivalently.
3. Unanswerable flag when corpus should abstain.
4. Effective date if policy questions are time-sensitive (vertical RAG).

Desk method note: we prefer dual annotation on 10–15% of items to measure label noise. Noisy labels make recall look worse than engineering reality—or hide real regressions.

Holdout and leakage control

• Separate dev (for chunk/embed sweeps) from holdout (for release gates).
• Never use holdout questions in few-shot prompts or judge tuning.
• When documents update, mark affected labels stale instead of silently wrong.

Step 3: Measure retrieval recall@k

Procedure

1. Freeze generator; disable reranker initially to isolate first-stage recall.
2. For each question, run retrieval with candidate k values (5, 10, 20, 50).
3. Mark hit if any gold chunk ID appears in results (respect ACL filters as in production).
4. Report recall@k overall and per product line / language slice.

Interpretation

Low recall@10 with high recall@50 suggests rerankers or smaller chunks may help. Low recall@50 suggests embedder mismatch, missing documents, bad chunk boundaries, or wrong metadata filters—not prompt engineering.

Ranking metrics

When multiple gold chunks exist or order matters for prompt assembly, compute MRR (mean reciprocal rank) or nDCG. Useful when comparing hybrid sparse+dense weightings beyond binary hit rate.

Step 4: Measure generation faithfulness

Human rubric (recommended anchor)

Sample 50–200 answers per release. Reviewers score:

• Fully supported: every factual claim appears in retrieved chunks.
• Partially supported: core claim supported; minor extrapolation.
• Unsupported: material claim absent from chunks.
• Abstention correct/incorrect for unanswerable cases.

LLM-as-judge (optional accelerator)

Calibrate judge prompts against human labels on dev set. Track judge–human agreement rate. If agreement drops after model swap, recalibrate before trusting automated faithfulness scores. Mis-calibrated judges reward fluent hallucinations— the same failure mode seen in public benchmark gaming discussions (leaderboard guide).

Citation spot checks

When UI shows footnotes, verify claim spans map to cited chunk text. Citation accuracy is a product metric users feel directly—especially in regulated verticals.

Step 5: End-to-end RAG QA scenarios

After retrieval and faithfulness pass component bars, run scenario tests:

• Multi-turn: follow-up pronouns resolving to prior retrieved entities.
• Tool + RAG agents: retrieval after tool output mutation (agent map).
• Memory hybrid: ensure durable facts come from index, not session drift (agent memory).
• Adversarial docs: injection attempts in corpus (safety pillar).

Step 6: Regression gates in CI/CD

1. On index or embedder change, run retrieval eval in staging.
2. Block promotion if recall@10 drops more than agreed threshold on holdout.
3. Run smoke faithfulness set on generator changes—even when retrieval unchanged.
4. Log config hash: embedder version, chunker version, k, filters, reranker model.

Store eval artifacts (question ID, retrieved IDs, scores) for diffing across releases. Debugging “what changed?” without artifacts wastes weeks.

Step 7: Anti-gaming and contamination

• Do not iterate exclusively on LLM-judge loops without human spot checks.
• Avoid training rerankers or embedders on exact holdout questions.
• Watch synthetic data contamination if gold sets are LLM-generated from the same docs without human review.
• Report metrics with confidence intervals when sample sizes are small per slice.

Public leaderboard contamination concerns apply privately too: if eval questions appear in fine-tune data, scores lie. Treat eval sets like secrets with access controls.

Failure modes and fixes

GraphRAG-specific eval extensions

Add cases requiring correct node/edge traversal: path recall (gold nodes present), constraint checks (role edges), and faithfulness of natural language to subgraph evidence. Vector metrics alone miss graph failures (GraphRAG explainer).

Reporting dashboard template

Publish weekly slices to stakeholders who do not read raw logs:

• Overall recall@5 / recall@10 vs prior week and vs release candidate.
• Faithfulness human sample rate and unsupported-claim rate.
• Top five failing question clusters (intent labels).
• Latency p50/p95 for embed, search, rerank, generate.
• Config hash: embedder, chunker, index version, generator ID.
• Abstention rate on unanswerable gold items (should be high and correct).

Synthetic data: use with caution

LLM-generated QA pairs from your docs can bootstrap labels quickly. Risks include:

• Questions that paraphrase headings too closely—inflated recall.
• Missed adversarial phrasing real users employ.
• Leakage if synthetic questions embed in fine-tune or judge prompts.

Desk practice: use synthetic data for dev sweeps only; promote questions to holdout only after human review and production mining.

Calibrating LLM-as-judge

1. Select 100 dev answers with human faithfulness labels.
2. Run candidate judge prompts; compute Cohen’s kappa or simple agreement rate.
3. Reject judges below agreed agreement threshold.
4. Re-calibrate after generator or judge model changes.
5. Keep human audit sample fixed size every release regardless of automated score.

Production monitoring without full re-label

• Citation click-through: Do users open cited sources?
• Thumbs down + reason codes: Tag retrieval vs generation vs policy.
• Escalation rate to human support after copilot answer.
• Empty retrieval rate: Queries returning zero chunks after filters.
• Drift detectors: Embedding distance shifts on sample queries week-over-week.

Multi-language and locale eval

Stratify metrics by locale. A strong English recall@10 can hide Spanish failures when users code-switch. Annotators must represent target locales; machine translation of gold sets without review mislabels evidence.

Versioning gold sets

Treat gold sets like code: semantic version, changelog, owners, review dates. When a label changes, record why (doc update vs annotation error). Downstream teams trust metrics only when label provenance is clear.

Connecting eval to stack decisions

Retrieval metrics justify investments in chunking and embedders (chunking guide). Faithfulness failures after generator upgrades belong in post-training / model selection conversations (model stack). Routing between retrain and retrieve should use eval evidence, not architecture fashion (RAG pillar).

Who this is for

• ML engineers and QA leads owning RAG release gates.
• Product managers defining SLOs for internal copilots.
• Data annotators and SMEs maintaining gold sets.

Who should skip

• Teams with no logging of chunk IDs—fix observability first.
• Organizations expecting one public MTEB score to replace task eval.
• Pre-production demos without representative questions—build gold skeleton early but do not overfit.

Common mistakes

• Evaluating only end-to-end user thumbs-up without retrieval decomposition.
• Using identical questions in prompts and holdout sets.
• Ignoring ACL filters in offline eval (inflated recall vs production).
• Refreshing docs without refreshing labels—silent metric drift.
• Chasing faithfulness with bigger models while recall@5 is 40%.

FAQ

How many labeled examples do we need?

Start with 200 covering top intents; expand when slice metrics stabilize and new product areas launch. Prefer breadth over perfect depth initially.

Is LLM-as-judge good enough alone?

No. Use it to scale after human calibration on a fixed dev set; keep periodic human audits on holdout.

Should we evaluate the embedder with MTEB only?

MTEB shortlists candidates; your recall@k on domain gold set decides deployment.

What recall@k target should we use?

Targets are product-specific. Internal copilots often aim for high recall@10 on critical intents before optimizing latency—define SLOs with stakeholders, not blog defaults.

What is a minimal eval for a v1 launch?

200 labeled questions, recall@10 gate, 50-answer human faithfulness sample, latency p95, and abstention checks on unanswerable items—expand slices after launch.

Worked week-one eval bootstrap

Day 1: Export 500 anonymized queries; dedupe and cluster intents. Day 2–3: SMEs label 200 with chunk IDs. Day 4: Script recall@10 against staging index. Day 5: Run generator; human-score 30 answers for faithfulness. Day 6: Fix top retrieval failure mode (usually chunking or hybrid). Day 7: Document config hash and ship gate thresholds. This beats months of prompt tuning without numbers.

Ragas and framework metrics (how to use without over-trusting)

Frameworks may expose faithfulness, answer relevance, context precision, and context recall proxies. Treat each as a screening signal:

• Run on dev only until correlated with human labels ≥ agreed threshold.
• Never optimize chunk size to maximize context precision alone if human faithfulness drops—metrics can disagree.
• Log raw judge outputs for failure analysis; aggregate scores hide span-level inventions.
• Pin framework and judge model versions in config hash alongside embedder and generator.

When frameworks update metric definitions between releases, re-baseline historical charts—do not compare v0.1 faithfulness to v0.2 without a bridge run.

Annotation playbook for SMEs

SMEs burn out on vague instructions. Provide:

• 10 worked examples of good vs bad gold chunk labels.
• Decision tree for unanswerable vs multi-chunk answers.
• Glossary linking product terms to doc section names.
• Weekly office hours for edge cases—not Slack threads without resolution.
• Inter-annotator agreement checks with adjudication sessions.

Quality beats quantity: 200 consistent labels outperform 800 noisy ones that teach the team to distrust metrics.

Release gate template (copy for PR descriptions)

• Config hash changed: yes/no (list components).
• Holdout recall@10 delta: +/− X pts (threshold ±Y).
• Faithfulness sample: N reviewed, unsupported rate Z%.
• Latency p95 delta within SLO: yes/no.
• Rollback index snapshot ID documented: yes/no.
• Human sign-off owner for regulated slices: name/role.

Metric definitions (quick reference)

Recall@k: fraction of questions where at least one gold chunk appears in top-k retrieved results.

MRR (mean reciprocal rank): average of 1/rank of the first gold chunk; rewards putting gold higher in the list.

nDCG: graded relevance when multiple gold chunks have different importance weights.

Faithfulness: fraction of answer claims supported by union of retrieved chunks under rubric rules—define “claim” granularity (sentence vs bullet) before scoring.

Context precision (framework sense): share of retrieved context tokens judged relevant—useful for diagnosing overweight k, not as sole north star.

When to split eval sets by product

Split when recall variance across products exceeds agreed threshold or when ACL models differ. A single aggregate recall@10 can hide a failing division that represents high regulatory exposure. Executives see rollup; engineers ship gates per slice.

Executive one-pager (what leadership should ask)

• Do we have holdout recall@10 and human faithfulness samples for this release?
• What config hash changed—embedder, chunker, or model?
• Which product slices regressed, if any?
• What is rollback time to prior index snapshot?
• Are regulated slices signed by named reviewers?

Leaders should not need to parse Ragas scores— they need clear gates and rollback stories. Engineering earns trust by saying “we blocked ship” when holdout failed, not by shipping and fixing live.

Continuous improvement loop

Monthly ritual (typical cadence): mine 50 new production failures; label retrieval vs generation cause; add 20 to gold set; pick top failure mode for engineering sprint; re-run holdout; publish dashboard delta to stakeholders. Quarterly: retire obsolete questions tied to deprecated products; refresh adversarial injection cases; re-calibrate LLM judge if generator version changed.

This loop connects eval to roadmap prioritization better than abstract “improve RAG” OKRs. It also prevents gold sets from becoming historical museums that no longer reflect user language after rebrands or feature renames.

Tooling landscape (non-exhaustive)

Teams combine custom Python scripts, search eval libraries, Ragas-style frameworks, and internal dashboards. No single tool replaces labeled data. Prefer tools that export per-question retrieval traces for debugging. Avoid frameworks that hide chunk IDs behind aggregate scores—debuggability matters more than polished UI in early maturity.

For CI integration, keep eval runtime under practical pipeline budgets: subset smoke eval on every PR; full holdout nightly. Long-running eval suites that engineers skip are worse than small honest smokes.

Aligning with model selection

When eval shows strong retrieval but weak faithfulness across generator candidates, run controlled generator shootouts on fixed retrieved contexts—isolate generator effect. Conversely, if faithfulness is high but answers incomplete, check whether k or chunk size truncates necessary evidence before blaming model IQ. This discipline connects eval to model stack conversations without conflating layers.

Documenting eval for audits

Auditors and clients increasingly ask how AI answers are tested. Maintain a living eval brief: scope of gold set, annotation process, metrics definitions, release gates, sampling plans for human review, and known limitations. Update version history when methodology changes. This document supports trust without claiming third-party certification.

Store eval artifacts with retention aligned to regulatory needs—ability to replay a historical answer’s retrieval trace months later ends many dispute cycles quickly.

Invite customer success and support leads to quarterly eval reviews—they see failure language users actually type, which SMEs sometimes sanitize out of gold sets. Their input keeps eval honest and prevents overfitting to polished internal phrasing.

When sharing eval dashboards externally, redact chunk text that contains MNPI or client identifiers—metrics can be public inside the firm while evidence stays scoped.

Eval maturity is a competitive advantage in regulated sales cycles—prospects increasingly ask for methodology, not demo sparkle. A clear eval brief can shorten security review even when raw metrics stay internal.

Pair eval milestones with leaderboard literacy training so executives do not override holdout gates based on unrelated public Elo scores.

Assign a named eval owner with budget for annotation—orphaned gold sets decay within two quarters as products rename features and docs move URLs. Ownership is cheaper than emergency re-labeling during a client audit.

Publish eval SLOs alongside latency SLOs in the same internal status page—teams treat what they see weekly. Hidden eval spreadsheets do not change shipping behavior when marketing deadlines press.

Quick reference card

Before blaming the LLM: check recall@10 on holdout with production ACL filters.

Before ship: config hash logged, faithfulness human sample reviewed, rollback snapshot ID ready.

Every quarter: mine new failures, expand gold set, recalibrate judge if generator changed.

Never: tune prompts for weeks without retrieval metrics; never share holdout questions in few-shot prompts.

Eval is the contract between ML, product, and leadership—write it down, version it, and review it on the same cadence as uptime reviews.

Start measuring today with imperfect labels; waiting for perfect annotation freezes teams in prompt-tuning loops that never converge.

The fastest team is not the one with the largest model—it is the one that knows whether retrieval failed before dinner.

Split retrieval and generation scores in every report—combined “RAG quality” numbers hide whether to hire search engineers or swap models.

Instrument first, argue later: a 200-question gold set with recall@10 and a 50-answer human faithfulness sample beats months of opinion about which embedder “feels better.”

If you cannot replay last week’s retrieval traces, you cannot debug last week’s incident—log chunk IDs from day one, even before the dashboard looks pretty.

Measure retrieval first; then faithfulness; then latency. That order prevents expensive generator swaps that never fix missing chunks.

Sources

1. Ragas documentation — reference metrics for RAG evaluation pipelines.
2. arXiv — RAG and faithfulness evaluation papers (verify specific citations per implementation).
3. Lewis et al., RAG (2020) — retrieval-generation baseline for metric design.

What we did not test: We did not publish a new composite RAG benchmark or run proprietary eval campaigns for this article. Metric recommendations follow public frameworks and desk synthesis—not EIA rankings.

Corrections: Update tooling names and metric definitions when frameworks change major versions; revise as-of date at top.

Next step

Apply eval patterns in regulated domains with vertical RAG: legal, medical, and finance, including human review gates and disclaimer UX.