Variance, sampling, and uncertainty
Replace one-run demos with repeated measurements, slices, confidence intervals, rare-failure hunting, and release evidence.
A new book by Jason Arbon
Testing AI
Engineering Confidence in Non-Deterministic Systems
The practical operating manual for people who have to decide whether AI-generated code, agents, models, and products are trustworthy enough to ship.
First edition. Amazon print and Kindle editions are being prepared.
21
98
42
32
The argument
Generation is easy. Validation is the hard part.
AI can produce more code, answers, designs, and decisions than any team can review the old way.
Most AI spending today still lives in experiments. The gap between an impressive demo and a production system is not usually another prompt trick. It is confidence: evidence that the system is useful, safe, economical, observable, reversible, and reliable enough for the people who depend on it.
Testing AI starts with testing because that is the doorway software teams recognize. It ends at confidence engineering: the operating discipline for shipping systems that vary from run to run, learn from changing data, call tools, personalize themselves, and behave differently under production load.
The book moves from first principles to operating practice: variance, sampling, statistical evidence, human and LLM judges, eval design, RAG, agents, generated code, security, observability, bias, interpretability, robotics, governance, and the future of AI testing itself.
Inside the book
A complete operating model for AI quality.
Not a catalog of tools. Not a stack of prompt recipes. The book connects measurement, engineering, operations, security, and judgment into one practical discipline.
Statistics that support decisions
Use p-values, effect sizes, power, paired tests, calibration, abstention, and practical significance without statistical theater.
Judges, raters, and evals
Design rubrics, calibrate humans and LLM judges, measure disagreement, manage labels, and build evals tied to product risk.
RAG, agents, and generated code
Test retrieval separately from generation, score tool trajectories, verify integration behavior, and catch code that looks right but is wrong.
Security, safety, and containment
Threat-model prompt injection, MCP permissions, poisoning, guardrails, dangerous capabilities, deception, and containment boundaries.
Operations and the future
Build observability, SLOs, canaries, incident response, cost controls, robotics simulation, governance, and continuous AI-led testing.
All 21 chapters
Part I
The New Shape of AI Quality
- 1The End of One-Run Testing
- 2From Tests to Release Evidence
- 3Sampling and Uncertainty
- 4Statistical Tests for AI Quality
- 5Judges, Humans, and Disagreement
Part II
Evidence, Evals, and Production Readiness
- 6Building Evals That Matter
- 7Release Readiness for AI Systems
- 8Operating AI: Observability, Relevance, and Economics
Part III
AI-Generated Software and Confidence Engineering
- 9Generated Code Changes the Job
- 10Anti-Patterns That Create False Confidence
- 11The Confidence Engineer
Part IV
Data, Security, Safety, and Model Internals
- 12Data, Bias, Raters, and Incentives
- 13AI Security and Guardrails
- 14Frontier Safety and Containment
- 15How Models Work
- 16Introspection: White-Box Testing Networks
Part V
Future Systems and the Practical Playbook
- 17Personalized and Dynamic AI Products
- 18Embodied and Long-Running AI Systems
- 19Governance, Regulation, and Moral Futures
- 20The Practical Playbook
- 21Predictions for the Tokenized Product Future
Who it is for
For everyone now responsible for confidence.
AI is collapsing the old walls between building, testing, product judgment, and operating software. The readers differ; the release decision is shared.
Developers and coding-agent users
Learn how to validate code and systems you did not fully write, including hidden integration, security, data, and operational failures.
Testers and automation engineers
Move beyond brittle exact assertions into evals, probabilistic evidence, production traces, risk slices, and human-machine review systems.
AI product builders and architects
Connect prompts, models, data, retrieval, tools, policies, user experience, cost, and release controls into one testable system.
Engineering leaders and executives
Ask for evidence that supports ship, hold, canary, rollback, investment, governance, and customer-risk decisions.
What changes after reading
You stop asking whether AI “works.”
You start asking the sharper questions that turn ambiguous behavior into engineering evidence.
- Which kinds of variation are harmless, and which are product failures?
- What population did we sample, and which risks are hidden by the average?
- Who or what is judging quality, and has that measurement system been calibrated?
- Did the system retrieve the right evidence, use tools safely, and preserve an auditable trace?
- Are quality gains worth their latency, compute cost, operational complexity, and user impact?
- What evidence justifies release, and what exact signal should trigger rollback?
Jason Arbon
Founder · AI quality builder · author
Founder · AI quality builder · author
Coming soon
Build faster. Fool yourself less.
Testing AI will be available on Amazon in print and Kindle editions. The purchase link will appear here as soon as the listing is live.