Section 084 · Chapter 10, Anti-Patterns That Create False Confidence
Anti-Patterns: Treating AI Bugs Like UI Bugs
Many AI failures do not have one screen, one selector, one line of code, or one obvious owner.
retrievaltreating bugs like ui bugs
What to do
- Define runnable checks that exercise retrieval and treating bugs like ui bugs.
- Set acceptable outcomes and blocker failures for retrieval and treating bugs like ui bugs before running the evaluation.
- Run representative cases for retrieval and treating bugs like ui bugs and preserve the failures that would change the decision.
Evidence to preserve
- Preserve the inputs, versions, configurations, raw outcomes, and results for retrieval, treating bugs like ui bugs needed to reproduce work on Anti-Patterns: Treating AI Bugs Like UI Bugs.
- Report results for retrieval, treating bugs like ui bugs by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
At scale, use AI incident templates with reproduction envelope, trace artifacts, affected slices, suspected contributors, severity, mitigation options, and post-mitigation eval results.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "Anti-Patterns: Treating AI Bugs Like UI Bugs." Testing AI Knowledge Edition, section 84.
https://jarbon.ai/testing-ai/knowledge/ch084-treating-bugs-like-ui-bugs.html