Section 020 · Chapter 4, Statistical Tests for AI Quality
Comparing Versions with t-tests
A t-test can help Confidence Engineers decide whether a difference in average scores is likely to be real or just sampling noise.
t-testp-valuestatistical significancepaired datacompare prompt or model versions
What to do
- Use judgment and risk analysis to decide whether the improvement is safe and worth shipping.
- Start with the data-generating process, not a favorite test.
- Ask what the experimental unit is, whether the same units saw both versions, what kind of outcome was measured, and which observations can influence one another.
- Compare versions within prompt, summarize or model the repeated runs, and calculate uncertainty at the prompt or cluster level.
Evidence to preserve
- Preserve the inputs, versions, configurations, raw outcomes, and results for t-test, p-value, statistical significance, paired data needed to reproduce work on Comparing Versions with t-tests.
- Report results for t-test, p-value, statistical significance, paired data by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
When the system matters, use paired tests when the same cases run through both versions. Pairing reduces noise from case difficulty. Also inspect assumptions: outliers, non-normal differences, multiple comparisons, and category-specific regressions can all make a tidy p-value misleading.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "Comparing Versions with t-tests." Testing AI Knowledge Edition, section 20.
https://jarbon.ai/testing-ai/knowledge/ch020-compare-versions-t-tests.html