Section 028 · Chapter 5, Judges, Humans, and Disagreement
LLM-as-a-Judge
LLM judges can make fuzzy evaluation faster, cheaper, and broader, but they need cost controls, calibration, disagreement review, and human oversight.
exact assertionsLLM judgerubric
What to do
- Use traditional testing techniques wherever they still work: exact assertions, schema checks, unit tests, component tests, static analysis, deterministic policy checks, citation-presence checks, and simple counters.
- Save LLM judges for the parts of quality that are genuinely semantic, fuzzy, contextual, or hard to express as deterministic code.
- Use deterministic checks first when the quality rule can be written down exactly.
- Use one calibrated LLM judge when the judgment is semantic and the risk is moderate.
- Use human review for severe failures, ambiguous cases, policy boundaries, and samples used to calibrate the judge.
Evidence to preserve
- Save LLM judges for the parts of quality that are genuinely semantic, fuzzy, contextual, or hard to express as deterministic code.
- Preserve the inputs, versions, configurations, raw outcomes, and results for exact assertions, LLM judge, rubric needed to reproduce work on LLM-as-a-Judge.
- Report results for exact assertions, LLM judge, rubric by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
Expert teams test the judge as a system under test. They measure agreement with human reviewers, track bias toward fluent answers, use blinded comparisons, keep judge prompts versioned, and quarantine examples where the judge reports low confidence or has historically been unreliable.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "LLM-as-a-Judge." Testing AI Knowledge Edition, section 28.
https://jarbon.ai/testing-ai/knowledge/ch028-llm-judge.html