Chapter 04
Statistical Tests for AI Quality
Choose the statistical test based on the data shape: paired vs independent, numeric vs categorical, ordinal vs binary. State the null hypothesis before running the eval. Report effect size and practical risk, not only p-values. Control false discoveries when trying many…
Apply this chapter
- Choose the statistical test based on the data shape: paired vs independent, numeric vs categorical, ordinal vs binary.
- State the null hypothesis before running the eval.
- Report effect size and practical risk, not only p-values.
- Control false discoveries when trying many prompts, models, policies, or slices.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
8 focused briefs
Concepts in this chapter
- 020Comparing Versions with t-testsA t-test can help Confidence Engineers decide whether a difference in average scores is likely to be real or just sampling noise.
- 021Null Hypothesis: What Are We Actually Testing?The null hypothesis is the boring default: assume the new AI system is not really better until the evidence is strong enough to challenge that…
- 022Chi-Squared Tests for Categorical AI QualityChi-squared tests help compare counts, categories, and failure distributions when quality is not a smooth 0-10 score.
- 023P-Values: Evidence, Not PermissionP-values can support a comparison, but they do not decide whether a product is safe, useful, or worth shipping.
- 024Statistical Significance vs. Practical SignificanceA difference can be statistically credible and still too small to matter. Confidence Engineers need to explain both sides.
- 025Power Analysis and Minimum Detectable EffectBefore asking whether a change won, builders should decide what size of win would actually matter.
- 026Multiple Comparisons and False DiscoveriesThe more slices, variants, and metrics you inspect, the more likely one lucky result will look real.
- 027F-Scores, Precision, Recall, and AI QualityF-scores help Confidence Engineers reason about the tradeoff between catching the right things and avoiding false alarms.