Section 003 · Chapter 1, The End of One-Run Testing
From Exact Assertions to Evaluation Criteria
When outputs can vary, builders need to move from brittle expected strings to clear properties that define acceptable behavior.
exact assertionsevaluation criteriarefusalcomplianceexact assertions evaluation criteria
What to do
- Define runnable checks that exercise exact assertions, evaluation criteria, and refusal.
- Set acceptable outcomes and blocker failures for exact assertions, evaluation criteria, and refusal before running the evaluation.
- Run representative cases for exact assertions, evaluation criteria, and refusal and preserve the failures that would change the decision.
Evidence to preserve
- Preserve the inputs, versions, configurations, raw outcomes, and results for exact assertions, evaluation criteria, refusal, compliance needed to reproduce work on From Exact Assertions to Evaluation Criteria.
- Report results for exact assertions, evaluation criteria, refusal, compliance by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
Expert teams usually split criteria into hard constraints and soft quality dimensions. Hard constraints are binary blockers, such as no private data leakage. Soft dimensions can be scored, such as clarity or completeness. Mixing the two into one score hides the failures that should stop release immediately.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "From Exact Assertions to Evaluation Criteria." Testing AI Knowledge Edition, section 3.
https://jarbon.ai/testing-ai/knowledge/ch003-exact-assertions-evaluation-criteria.html