Section 093 · Chapter 12, Data, Bias, Raters, and Incentives

Bias Taxonomy for AI Systems

You cannot test bias well until you name which kind of bias you are looking for.

bias taxonomy

What to do

Define runnable checks that exercise bias taxonomy.
Set acceptable outcomes and blocker failures for bias taxonomy before running the evaluation.
Run representative cases for bias taxonomy and preserve the failures that would change the decision.

Evidence to preserve

Preserve the inputs, versions, configurations, raw outcomes, and results for bias taxonomy needed to reproduce work on Bias Taxonomy for AI Systems.
Report results for bias taxonomy by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.

Expert note

At scale, create a bias risk taxonomy for the product domain, then map each bias type to eval slices, counterfactual tests, raters, severity labels, and mitigation owners. Bias testing should be domain-specific, not a generic checkbox.

Continue the conversation

Apply this to your context.

Save your product context once, then open a focused conversation that combines it with this concept.

Cite this page

Jason Arbon. "Bias Taxonomy for AI Systems." Testing AI Knowledge Edition, section 93.

https://jarbon.ai/testing-ai/knowledge/ch093-bias-taxonomy.html