Section 172 · Chapter 20, The Practical Playbook
Minimum Viable AI Quality System
If the book feels large, start here: a small quality system that produces real evidence instead of ritual.
benchmarkdeceptionminimum viable quality
What to do
- Start with roughly fifty production-shaped eval cases.
- Use a 0-10 or 0-1 score, but define what the numbers mean.
- Use one LLM judge for scale, but calibrate it against human review.
- Review disagreements instead of hiding them.
- Log traces for every run: prompt, model, system message, retrieval context, tool calls, tool arguments, response, cost, latency, judge version, rubric version, and final decision.
Evidence to preserve
- Include common use, high-value business flows, edge cases, policy boundaries, security-sensitive cases, confusing user inputs, and a few known failures from production or dogfooding.
- Include hard blockers for privacy leaks, unsafe tool calls, policy violations, severe hallucinations, irreversible actions, and failures that would embarrass the company if screenshotted.
- Log traces for every run: prompt, model, system message, retrieval context, tool calls, tool arguments, response, cost, latency, judge version, rubric version, and final decision.
- Preserve the inputs, versions, configurations, raw outcomes, and results for benchmark, deception, minimum viable quality needed to reproduce work on Minimum Viable AI Quality System.
Expert note
In a real release review, treat the minimum system as an evolving control system. Version the cases, rubric, judge, model, prompts, policies, retrieval index, tools, and release thresholds together. A score without provenance is not evidence.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "Minimum Viable AI Quality System." Testing AI Knowledge Edition, section 172.
https://jarbon.ai/testing-ai/knowledge/ch172-minimum-viable-quality.html