Section 073 · Chapter 10, Anti-Patterns That Create False Confidence

Anti-Patterns: Over-Specific Test Plans and Test Cases

Exact steps and exact expected words can make AI tests brittle while missing the behavior that matters.

over specific test plans cases

What to do

Define what the user is trying to accomplish, what must be true, what must never happen, and how quality will be judged.
Use rubrics, properties, metamorphic relationships, schemas, blocker rules, and examples of acceptable variation.
Keep exact assertions for things that must be exact, such as JSON shape, policy-required language, citations, and irreversible-action confirmations.
Use exact checks for contracts and safety boundaries, and rubrics or judge-scored properties for open-ended behavior.

Evidence to preserve

Preserve the inputs, versions, configurations, raw outcomes, and results for over specific test plans cases needed to reproduce work on Anti-Patterns: Over-Specific Test Plans and Test Cases.
Report results for over specific test plans cases by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.

Expert note

In production work, separate hard invariants from soft preferences. Use exact checks for contracts and safety boundaries, and rubrics or judge-scored properties for open-ended behavior.

Continue the conversation

Apply this to your context.

Save your product context once, then open a focused conversation that combines it with this concept.

Cite this page

Jason Arbon. "Anti-Patterns: Over-Specific Test Plans and Test Cases." Testing AI Knowledge Edition, section 73.

https://jarbon.ai/testing-ai/knowledge/ch073-over-specific-test-plans-cases.html