Section 072 · Chapter 10, Anti-Patterns That Create False Confidence
Anti-Patterns: Percent Passed Is Not Quality
A 94% pass rate can be comforting, meaningless, or dangerous depending on what failed.
quality metricpercent passed quality
What to do
- Define runnable checks that exercise quality metric and percent passed quality.
- Set acceptable outcomes and blocker failures for quality metric and percent passed quality before running the evaluation.
- Run representative cases for quality metric and percent passed quality and preserve the failures that would change the decision.
Evidence to preserve
- Preserve the inputs, versions, configurations, raw outcomes, and results for quality metric, percent passed quality needed to reproduce work on Anti-Patterns: Percent Passed Is Not Quality.
- Report results for quality metric, percent passed quality by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
In a real release review, pair pass rate with severity-adjusted score, blocker rate, confidence interval, slice-level minimums, and dataset composition. Trend pass rate only when the test population and scoring rules are comparable.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "Anti-Patterns: Percent Passed Is Not Quality." Testing AI Knowledge Edition, section 72.
https://jarbon.ai/testing-ai/knowledge/ch072-percent-passed-quality.html