Section 075 · Chapter 10, Anti-Patterns That Create False Confidence

Anti-Patterns: Filing Every Bad Output Like a Bug

One bad AI output is usually evidence of a behavior pattern, not a single defect with a surgical fix.

filing bad output like bug

What to do

Define runnable checks that exercise filing bad output like bug.
Set acceptable outcomes and blocker failures for filing bad output like bug before running the evaluation.
Run representative cases for filing bad output like bug and preserve the failures that would change the decision.

Evidence to preserve

Preserve the inputs, versions, configurations, raw outcomes, and results for filing bad output like bug needed to reproduce work on Anti-Patterns: Filing Every Bad Output Like a Bug.
Report results for filing bad output like bug by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.

Expert note

AI issue tracking should include cluster id, slice, severity, sample count, confidence, regression cases, mitigation hypothesis, and post-fix distribution movement. The fix is not done when one example disappears.

Continue the conversation

Apply this to your context.

Save your product context once, then open a focused conversation that combines it with this concept.

Cite this page

Jason Arbon. "Anti-Patterns: Filing Every Bad Output Like a Bug." Testing AI Knowledge Edition, section 75.

https://jarbon.ai/testing-ai/knowledge/ch075-filing-bad-output-like-bug.html