RAG evaluation and failure analysis

RAG quality has at least two distinct parts. Retrieval metrics ask whether relevant evidence is found and ranked highly enough; answer metrics ask whether the response is correct, grounded, complete, and properly cited. Build examples with known supporting sources, unanswerable questions, ambiguous wording, and stale or conflicting documents. Keep the corpus snapshot, query set, prompt, model, and retrieval settings versioned for comparisons. When a response fails, inspect the pipeline in order: was the source ingested, did chunking preserve the relevant passage, did filters allow it, did retrieval return it, did context construction include it, and did generation use it correctly? Fix the earliest failing step, then rerun regression examples. A RAG evaluation needs separate evidence for retrieval and response quality. For retrieval, label relevant source IDs and inspect recall at the candidate cutoff and ranking of the best evidence. For generation, score correctness, groundedness, citation support, completeness, and appropriate abstention. Keep the source snapshot and query set stable while comparing changes. When an answer fails, classify the earliest cause: source not ingested, chunk lost context, filter removed it, ranker missed it, prompt omitted it, or model misused it. This diagnosis prevents fixing a retrieval defect by adding more prompt text. Implementation drill: trace one representative request through the stages in the diagram. Write down what enters each stage, what output or evidence it produces, and which identity is allowed to perform it. Then test an empty input, an unauthorized request, a dependency timeout, and an unexpected result. For each case, state whether the application should retry, reject, abstain, or ask a person for help. Keep the observed failure as a regression example so changes to prompts, models, data, or code can be checked against the same behavior. Practice task: What is the best first step when an answer lacks a relevant source?