Section 022 · Chapter 4, Statistical Tests for AI Quality
Chi-Squared Tests for Categorical AI Quality
Chi-squared tests help compare counts, categories, and failure distributions when quality is not a smooth 0-10 score.
What to do
- Use chi-squared tests when the outcome is categorical: pass or fail, safe or unsafe, grounded or hallucinated, answer or refuse, correct tool or wrong tool, satisfied or escalated.
- Report counts and percentages, not only p-values.
- Define runnable checks that exercise chi-squared and chi squared tests categorical quality.
Evidence to preserve
- Report counts and percentages, not only p-values.
- Preserve the inputs, versions, configurations, raw outcomes, and results for chi-squared, chi squared tests categorical quality needed to reproduce work on Chi-Squared Tests for Categorical AI Quality.
- Report results for chi-squared, chi squared tests categorical quality by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
Check whether the test assumptions fit the data. Chi-squared tests expect independent observations and enough expected count in each cell. If expected counts are tiny, use Fisher's exact test or combine categories carefully before testing. Fisher's exact test does not require knowing what changed inside a black-box AI system. It only needs the observed counts and a valid comparison design. The hard part is not opening the model; the hard part is making sure the cases, categories, and sampling process are meaningful.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "Chi-Squared Tests for Categorical AI Quality." Testing AI Knowledge Edition, section 22.
https://jarbon.ai/testing-ai/knowledge/ch022-chi-squared-tests-categorical-quality.html