Chapter 02
From Tests to Release Evidence
Turn tests into release evidence: cases, slices, traces, reviewer decisions, and explicit gates. Add metamorphic checks where equivalent inputs should preserve important behavior. Use risk-based and stratified sampling so high-impact slices are not buried by easy cases. Log…
Apply this chapter
- Turn tests into release evidence: cases, slices, traces, reviewer decisions, and explicit gates.
- Add metamorphic checks where equivalent inputs should preserve important behavior.
- Use risk-based and stratified sampling so high-impact slices are not buried by easy cases.
- Log enough to replay failures: prompt, context, model, tools, retrieved evidence, versions, scores, and reviewer rationale.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
8 focused briefs
Concepts in this chapter
- 007Metamorphic TestingWhen there is no single correct answer, builders can change the input and check whether important relationships still hold.
- 008Golden Sets and Live SamplingStable regression examples and fresh real-world samples solve different problems. Mature AI testing needs both.
- 009Risk-Based SamplingTesting effort should follow risk. High-impact failures deserve more samples, stricter gates, and deeper review.
- 010Stratified ReportingOverall averages can hide weak segments. Break results down by the categories that matter.
- 011Rare Failure HuntingAverage quality can look excellent while rare catastrophic failures still make the system unsafe.
- 012Pairwise ComparisonWhen absolute scoring is hard, asking which output is better can produce useful evidence.
- 013Reproducibility: Logging the Right ThingsNon-deterministic bugs are hard to debug unless Confidence Engineers capture the context around the failure.
- 014Release Gates for Non-Deterministic SystemsA good release gate combines average quality, uncertainty, failure rates, hard safety rules, and category-specific risk.