Section 005 · Chapter 1, The End of One-Run Testing
Variance: Not All Differences Are Bugs
Good testing distinguishes harmless variation from variation that changes facts, safety, reliability, or user trust.
variancevariance differences bugs
What to do
- Define that boundary before testing.
- Report variance by dimension.
- Score variance, factual variance, latency variance, ranking variance, policy variance, and action variance are not interchangeable.
Evidence to preserve
- Report variance by dimension.
- Preserve the inputs, versions, configurations, raw outcomes, and results for variance, variance differences bugs needed to reproduce work on Variance: Not All Differences Are Bugs.
- Report results for variance, variance differences bugs by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
Report variance by dimension. Score variance, factual variance, latency variance, ranking variance, policy variance, and action variance are not interchangeable. A single average can hide the type of instability users will actually feel.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "Variance: Not All Differences Are Bugs." Testing AI Knowledge Edition, section 5.
https://jarbon.ai/testing-ai/knowledge/ch005-variance-differences-bugs.html