Section 067 · Chapter 9, Generated Code Changes the Job
AI-Generated Tests and the Illusion of Coverage
AI-generated tests can raise coverage numbers while failing to catch the bugs that matter.
RAGgenerated tests illusion coverage
What to do
- Use AI to generate test ideas, but ask it for adversarial cases, boundary cases, and property-based cases, not just straightforward unit tests.
- Use mutation testing, requirement coverage, negative-case coverage, contract tests, and historical defect replay.
- Define runnable checks that exercise RAG and generated tests illusion coverage.
Evidence to preserve
- Preserve the inputs, versions, configurations, raw outcomes, and results for RAG, generated tests illusion coverage needed to reproduce work on AI-Generated Tests and the Illusion of Coverage.
- Report results for RAG, generated tests illusion coverage by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
The deeper move is to score tests by fault-detection power. Use mutation testing, requirement coverage, negative-case coverage, contract tests, and historical defect replay. AI-generated tests should be reviewed as critically as AI-generated production code.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "AI-Generated Tests and the Illusion of Coverage." Testing AI Knowledge Edition, section 67.
https://jarbon.ai/testing-ai/knowledge/ch067-generated-tests-illusion-coverage.html