Chapter 03
Sampling and Uncertainty
Estimate behavior from samples without pretending the sample is the truth. Report confidence intervals and sample counts next to scores. Prefer paired comparisons when the same cases run through competing versions. Separate model-reported confidence from measured statistical…
Apply this chapter
- Estimate behavior from samples without pretending the sample is the truth.
- Report confidence intervals and sample counts next to scores.
- Prefer paired comparisons when the same cases run through competing versions.
- Separate model-reported confidence from measured statistical confidence.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
5 focused briefs
Concepts in this chapter
- 015Sampling: One Run Tells You Almost NothingFor unpredictable systems, a single output is an anecdote. A sample is the beginning of evidence.
- 016How Many Samples Are Enough?Sample size is a risk decision. The higher the stakes and the rarer the failure, the more evidence builders need.
- 017Basic Stats Every AI Builder Should KnowA few practical statistics can help developers explain non-deterministic quality without pretending the data is more precise than it is.
- 018Confidence Intervals: Saying "About" Like a ProfessionalConfidence intervals help Confidence Engineers report estimates as ranges instead of pretending sample results are exact truth.
- 019AI-Reported Confidence vs. Statistical ConfidenceAn LLM saying it is confident is not the same as a confidence interval calculated from sample data.