Chapter 05
Judges, Humans, and Disagreement
Define the human measurement system before automating it with an LLM judge. Write rubrics with concrete evidence requirements and calibration examples. Measure disagreement instead of hiding it; disagreement may signal ambiguity, missing context, or useful diversity. Calibrate…
Apply this chapter
- Define the human measurement system before automating it with an LLM judge.
- Write rubrics with concrete evidence requirements and calibration examples.
- Measure disagreement instead of hiding it; disagreement may signal ambiguity, missing context, or useful diversity.
- Calibrate LLM judges against humans and discount or abstain when the judge is weak.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
8 focused briefs
Concepts in this chapter
- 028LLM-as-a-JudgeLLM judges can make fuzzy evaluation faster, cheaper, and broader, but they need cost controls, calibration, disagreement review, and human…
- 029Human Calibration of LLM JudgesBefore relying on an LLM judge at scale, builders need to know whether it scores like a trusted human reviewer.
- 030Inter-Rater AgreementWhen reviewers disagree often, the evaluation system may need as much attention as the product being evaluated.
- 031Disagreement, Diversity, and Topical EntropyHuman and AI disagreement is not always a defect. Sometimes it is a signal that different users value different good answers.
- 032Rubrics That Actually WorkA good rubric turns fuzzy judgment into repeatable evaluation. A bad rubric creates fake precision.
- 033Using Raters WellHuman raters are not a checkbox. They are an evaluation instrument that needs selection, calibration, workflow design, and quality control.
- 034Testing the Value of Data LabelersData labelers create the ground truth many AI evaluations depend on. Their value should be measured, not assumed.
- 035Data Labeling Dangers and Labeler DemographicsThe people and systems that create labels become part of the product's definition of quality.