Section 015 · Chapter 3, Sampling and Uncertainty

Sampling: One Run Tells You Almost Nothing

For unpredictable systems, a single output is an anecdote. A sample is the beginning of evidence.

confidence engineersampling tells you almost nothing

What to do

Track slices for typos, dialects, accents, code-switching, non-native phrasing, messy intent, and other realistic input forms so the team can see where quality actually breaks.
Define runnable checks that exercise confidence engineer and sampling tells you almost nothing.
Set acceptable outcomes and blocker failures for confidence engineer and sampling tells you almost nothing before running the evaluation.

Evidence to preserve

Track slices for typos, dialects, accents, code-switching, non-native phrasing, messy intent, and other realistic input forms so the team can see where quality actually breaks.
Preserve the inputs, versions, configurations, raw outcomes, and results for confidence engineer, sampling tells you almost nothing needed to reproduce work on Sampling: One Run Tells You Almost Nothing.
Report results for confidence engineer, sampling tells you almost nothing by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.

Expert note

Expert sampling plans specify the population, sampling frame, inclusion criteria, exclusions, randomization method, known bias, and input-variation strategy. If the sample only includes easy happy-path prompts, the confidence interval describes easy happy-path prompts, not the product. Track slices for typos, dialects, accents, code-switching, non-native phrasing, messy intent, and other realistic input forms so the team can see where quality actually breaks.

Continue the conversation

Apply this to your context.

Save your product context once, then open a focused conversation that combines it with this concept.

Cite this page

Jason Arbon. "Sampling: One Run Tells You Almost Nothing." Testing AI Knowledge Edition, section 15.

https://jarbon.ai/testing-ai/knowledge/ch015-sampling-tells-you-almost-nothing.html