Section 080 · Chapter 10, Anti-Patterns That Create False Confidence
Anti-Patterns: Testing Only the Final Answer
For RAG and agents, the visible answer is only the last step in a larger system.
RAGonly final answer
What to do
- Inspect the path: retrieved documents, tool calls, intermediate observations, decisions, and final output.
- Score retrieval, planning, tool choice, arguments, permission boundaries, recovery, final answer, and side effects separately.
- Define runnable checks that exercise RAG and only final answer.
Evidence to preserve
- Preserve the inputs, versions, configurations, raw outcomes, and results for RAG, only final answer needed to reproduce work on Anti-Patterns: Testing Only the Final Answer.
- Report results for RAG, only final answer by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
In a real release review, store traces as eval artifacts. Score retrieval, planning, tool choice, arguments, permission boundaries, recovery, final answer, and side effects separately.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "Anti-Patterns: Testing Only the Final Answer." Testing AI Knowledge Edition, section 80.
https://jarbon.ai/testing-ai/knowledge/ch080-only-final-answer.html