Section 012 · Chapter 2, From Tests to Release Evidence
Pairwise Comparison
When absolute scoring is hard, asking which output is better can produce useful evidence.
pairwise comparison
What to do
- Review disagreement cases, especially when the judge chooses a fluent but factually weaker answer.
- Version B adds a deterministic fake clock and asserts that exactly three retries happen before the checkout flow gives up.
- Version A hides the timing problem and makes the suite slower.
- Version B explains the behavior and makes the test more stable.
Evidence to preserve
- Preserve the inputs, versions, configurations, raw outcomes, and results for pairwise comparison needed to reproduce work on Pairwise Comparison.
- Report results for pairwise comparison by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
Blind the version labels, randomize side order, allow ties, and analyze win rate by category. Pairwise wins do not replace absolute gates because the better of two bad outputs can still be unacceptable.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "Pairwise Comparison." Testing AI Knowledge Edition, section 12.
https://jarbon.ai/testing-ai/knowledge/ch012-pairwise-comparison.html