Section 078 · Chapter 10, Anti-Patterns That Create False Confidence
Anti-Patterns: The Static Test Plan
A frozen test plan can look responsible while the AI system keeps changing underneath it.
retrievalstatic test plan
What to do
- Define runnable checks that exercise retrieval and static test plan.
- Set acceptable outcomes and blocker failures for retrieval and static test plan before running the evaluation.
- Run representative cases for retrieval and static test plan and preserve the failures that would change the decision.
Evidence to preserve
- Preserve the inputs, versions, configurations, raw outcomes, and results for retrieval, static test plan needed to reproduce work on Anti-Patterns: The Static Test Plan.
- Report results for retrieval, static test plan by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
The deeper move is to maintain a living quality system: versioned evals, changelogs, production trace mining, drift monitors, rubric updates, and explicit compatibility rules for trend comparisons.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "Anti-Patterns: The Static Test Plan." Testing AI Knowledge Edition, section 78.
https://jarbon.ai/testing-ai/knowledge/ch078-static-test-plan.html