Section 052 · Chapter 8, Operating AI: Observability, Relevance, and Economics
RAG Evaluation
RAG systems fail in two places: what they retrieve and what they say with it.
RAGretrievalgroundednesscitation faithfulnesscontext precisioncontext recall
What to do
- Start with retrieval quality.
- Track retrieval hit rate, context precision, context recall, freshness, duplicate chunks, and whether the top results contain the needed evidence.
- Run four versions of the case: current documentation ranked first, current documentation buried, conflicting versions retrieved together, and current documentation missing entirely.
- Track context precision, context recall, retrieval hit rate, chunk freshness, reranker quality, answer faithfulness, citation support, abstention behavior, and failure attribution by document source and query class.
Evidence to preserve
- Track retrieval hit rate, context precision, context recall, freshness, duplicate chunks, and whether the top results contain the needed evidence.
- Track context precision, context recall, retrieval hit rate, chunk freshness, reranker quality, answer faithfulness, citation support, abstention behavior, and failure attribution by document source and query class.
- Preserve the inputs, versions, configurations, raw outcomes, and results for RAG, retrieval, groundedness, citation faithfulness needed to reproduce work on RAG Evaluation.
- Report results for RAG, retrieval, groundedness, citation faithfulness by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
In production work, separate retriever metrics from generator metrics. Track context precision, context recall, retrieval hit rate, chunk freshness, reranker quality, answer faithfulness, citation support, abstention behavior, and failure attribution by document source and query class.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "RAG Evaluation." Testing AI Knowledge Edition, section 52.
https://jarbon.ai/testing-ai/knowledge/ch052-evaluate-rag.html