Section 063 · Chapter 9, Generated Code Changes the Job
AI-Generated Code That Looks Right but Is Wrong
AI-generated code often fails in a dangerous way: it looks clean, compiles, and still implements the wrong behavior.
generated codegenerated code looks right wrong
What to do
- Do not only test the happy path the prompt described.
- Test the neighboring cases the prompt did not mention.
- Use property-based tests, metamorphic tests, boundary matrices, and requirement-to-test traceability.
Evidence to preserve
- Preserve the inputs, versions, configurations, raw outcomes, and results for generated code, generated code looks right wrong needed to reproduce work on AI-Generated Code That Looks Right but Is Wrong.
- Report results for generated code, generated code looks right wrong by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
Use property-based tests, metamorphic tests, boundary matrices, and requirement-to-test traceability. AI-generated code should be judged by behavioral evidence, not by whether it looks idiomatic.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "AI-Generated Code That Looks Right but Is Wrong." Testing AI Knowledge Edition, section 63.
https://jarbon.ai/testing-ai/knowledge/ch063-generated-code-looks-right-wrong.html