Testing AI

A new book by Jason Arbon

Testing AI

Engineering Confidence in Non-Deterministic Systems

The practical operating manual for people who have to decide whether AI-generated code, agents, models, and products are trustworthy enough to ship.

Buy on Amazon Available now

First edition. Available now on Amazon in print and Kindle editions.

The argument

Generation is easy. Validation is the hard part.

AI can produce more code, answers, designs, and decisions than any team can review the old way.

Most AI spending today still lives in experiments. The gap between an impressive demo and a production system is not usually another prompt trick. It is confidence: evidence that the system is useful, safe, economical, observable, reversible, and reliable enough for the people who depend on it.

Testing AI starts with testing because that is the doorway software teams recognize. It ends at confidence engineering: the operating discipline for shipping systems that vary from run to run, learn from changing data, call tools, personalize themselves, and behave differently under production load.

The book moves from first principles to operating practice: variance, sampling, statistical evidence, human and LLM judges, eval design, RAG, agents, generated code, security, observability, bias, interpretability, robotics, governance, and the future of AI testing itself.

Inside the book

A complete operating model for AI quality.

Not a catalog of tools. Not a stack of prompt recipes. The book connects measurement, engineering, operations, security, and judgment into one practical discipline.

Variance, sampling, and uncertainty

Replace one-run demos with repeated measurements, slices, confidence intervals, rare-failure hunting, and release evidence.

Statistics that support decisions

Use p-values, effect sizes, power, paired tests, calibration, abstention, and practical significance without statistical theater.

Judges, raters, and evals

Design rubrics, calibrate humans and LLM judges, measure disagreement, manage labels, and build evals tied to product risk.

RAG, agents, and generated code

Test retrieval separately from generation, score tool trajectories, verify integration behavior, and catch code that looks right but is wrong.

Security, safety, and containment

Threat-model prompt injection, MCP permissions, poisoning, guardrails, dangerous capabilities, deception, and containment boundaries.

Operations and the future

Build observability, SLOs, canaries, incident response, cost controls, robotics simulation, governance, and continuous AI-led testing.

All 21 chapters

Part I

The New Shape of AI Quality

  1. 1The End of One-Run Testing
  2. 2From Tests to Release Evidence
  3. 3Sampling and Uncertainty
  4. 4Statistical Tests for AI Quality
  5. 5Judges, Humans, and Disagreement
Part II

Evidence, Evals, and Production Readiness

  1. 6Building Evals That Matter
  2. 7Release Readiness for AI Systems
  3. 8Operating AI: Observability, Relevance, and Economics
Part III

AI-Generated Software and Confidence Engineering

  1. 9Generated Code Changes the Job
  2. 10Anti-Patterns That Create False Confidence
  3. 11The Confidence Engineer
Part IV

Data, Security, Safety, and Model Internals

  1. 12Data, Bias, Raters, and Incentives
  2. 13AI Security and Guardrails
  3. 14Frontier Safety and Containment
  4. 15How Models Work
  5. 16Introspection: White-Box Testing Networks
Part V

Future Systems and the Practical Playbook

  1. 17Personalized and Dynamic AI Products
  2. 18Embodied and Long-Running AI Systems
  3. 19Governance, Regulation, and Moral Futures
  4. 20The Practical Playbook
  5. 21Predictions for the Tokenized Product Future

Every chapter is also available as a short, mobile-first PDF carousel. Read it in your browser or download it to share on LinkedIn.

01
The End of One-Run TestingOpen PDF carousel
 02
From Tests to Release EvidenceOpen PDF carousel
 03
Sampling and UncertaintyOpen PDF carousel
 04
Statistical Tests for AI QualityOpen PDF carousel
 05
Judges, Humans, and DisagreementOpen PDF carousel
 06
Building Evals That MatterOpen PDF carousel
 07
Release Readiness for AI SystemsOpen PDF carousel
 08
Operating AIOpen PDF carousel
 09
Generated Code Changes the JobOpen PDF carousel
 10
Anti-Patterns That Create False ConfidenceOpen PDF carousel
 11
The Confidence EngineerOpen PDF carousel
 12
Data, Bias, Raters, and IncentivesOpen PDF carousel
 13
AI Security and GuardrailsOpen PDF carousel
 14
Frontier Safety and ContainmentOpen PDF carousel
 15
How Models WorkOpen PDF carousel
 16
Introspection: White-Box Testing NetworksOpen PDF carousel
 17
Personalized and Dynamic AI ProductsOpen PDF carousel
 18
Embodied and Long-Running AI SystemsOpen PDF carousel
 19
Governance, Regulation, and Moral FuturesOpen PDF carousel
 20
The Practical PlaybookOpen PDF carousel
 21
Predictions for the Tokenized Product FutureOpen PDF carousel

Companion skills

Install the book as coding-agent habits.

The public skills are now intentionally small: one overall Testing AI skill, plus one actionable skill for each narrative chapter. Each chapter skill has distinct trigger vocabulary and tells an AI coding agent what evidence, tests, traces, and release artifacts to produce.

Testing AI skills for coding agents

Use the overall skill when you want the full confidence-engineering workflow. Use a chapter skill when the work is specific: sampling, statistical tests, judges, RAG, generated code, security, robotics, governance, or the future of AI testing AI.

Open the GitHub repo Open the overall book skill
Chapter skills
  1. Ch 01The End of One-Run Testing
  2. Ch 02From Tests to Release Evidence
  3. Ch 03Sampling and Uncertainty
  4. Ch 04Statistical Tests for AI Quality
  5. Ch 05Judges, Humans, and Disagreement
  6. Ch 06Building Evals That Matter
  7. Ch 07Release Readiness for AI Systems
  8. Ch 08Operating AI: Observability, Relevance, and Economics
  9. Ch 09Generated Code Changes the Job
  10. Ch 10Anti-Patterns That Create False Confidence
  11. Ch 11The Confidence Engineer
  12. Ch 12Data, Bias, Raters, and Incentives
  13. Ch 13AI Security and Guardrails
  14. Ch 14Frontier Safety and Containment
  15. Ch 15How Models Work
  16. Ch 16Introspection: White-Box Testing Networks
  17. Ch 17Personalized and Dynamic AI Products
  18. Ch 18Embodied and Long-Running AI Systems
  19. Ch 19Governance, Regulation, and Moral Futures
  20. Ch 20The Practical Playbook
  21. Ch 21Predictions for the Tokenized Product Future

Who it is for

For everyone now responsible for confidence.

AI is collapsing the old walls between building, testing, product judgment, and operating software. The readers differ; the release decision is shared.

Developers and coding-agent users

Learn how to validate code and systems you did not fully write, including hidden integration, security, data, and operational failures.

Testers and automation engineers

Move beyond brittle exact assertions into evals, probabilistic evidence, production traces, risk slices, and human-machine review systems.

AI product builders and architects

Connect prompts, models, data, retrieval, tools, policies, user experience, cost, and release controls into one testable system.

Engineering leaders and executives

Ask for evidence that supports ship, hold, canary, rollback, investment, governance, and customer-risk decisions.

What changes after reading

You stop asking whether AI “works.”

You start asking the sharper questions that turn ambiguous behavior into engineering evidence.

  1. Which kinds of variation are harmless, and which are product failures?
  2. What population did we sample, and which risks are hidden by the average?
  3. Who or what is judging quality, and has that measurement system been calibrated?
  4. Did the system retrieve the right evidence, use tools safely, and preserve an auditable trace?
  5. Are quality gains worth their latency, compute cost, operational complexity, and user impact?
  6. What evidence justifies release, and what exact signal should trigger rollback?

Draft reviewers

Thank you to the early readers who made the book better.

This book improved because people took time to read rough chapters, mark confusing passages, suggest missing topics, and push the material toward practical usefulness for the broader AI quality community.

Karime Salomon Zarate

Missing topics, definitions, chatbot quality criteria, invisible dependency failures, editorial clarity, and practical reviewer perspective.

Jeffery Evans

Regulatory stakes, ISO/IEC 42001 context, beginner-friendly SKILL.md explanation, reader navigation, sampling definitions, p-value clarity, and release-gate examples.

Gastón Marichal

Non-deterministic framing, varying degrees of correctness and failure, and clearer calibration language.

Christopher L.

Editorial clarity around confidence-interval language and a more beginner-friendly explanation of how intervals are calculated.

Richard Hill

Quality dimensions, sample-size nuance, rare-failure hunting, AI judge bias questions, and practical evaluator/tooling suggestions.

Andre Hulse

Front-matter positioning, preface structure, reader framing, and a clearer bridge from traditional software quality into AI quality.

Steven Knopf

Ask AI link behavior, dense-sentence cleanup, link hygiene, repeated-example spotting, and draft-reader experience.

Kiran Rayachoti

The suggestion to treat performance engineering as a named AI quality discipline rather than a narrow testing activity.

Sergio Segura

LLM-as-a-judge cost realism, practical eval tooling, and multi-judge voting nuance for high-stakes evaluation.

Zoltán Tarkó

Senior-engineering feedback on tone, accountable AI builders, pass/fail nuance, judge confidence wording, and LLM-judge consistency.

Larry Petersen

Terminology consistency around testers, quality engineers, and Confidence Engineers.

Jason Arbon
Jason Arbon
Founder · AI quality builder · author

About the author

Jason has spent his career ensuring the quality of large, complex, and non-deterministic systems.

Jason Arbon is an entrepreneur, co-author of How Google Tests Software, and founder of AI testing companies including test.ai and Testers.ai. He has built and led quality systems across Microsoft, Bing, Google Search, Chrome, ChromeOS, Applause, and AI-first testing products.

His work sits at the intersection of software engineering, product judgment, statistics, human evaluation, and autonomous testing. Testing AI brings those experiences together into Confidence Engineering: the discipline of building justified confidence in software created by and with AI.

LinkedIn Jason's projects Testers.ai Writing

Also by Jason Arbon

Two earlier books about quality at scale.

The new book extends a long-running argument: quality becomes useful when teams turn it into a system, not a final inspection.

Cover of How Google Tests Software
Published

How Google Tests Software

Co-authored with James Whittaker and Jeff Carollo. An inside account of testing roles, infrastructure, and engineering culture at Google scale.

View on Amazon
Cover of App Quality: Secrets for Agile App Teams
Published

App Quality: Secrets for Agile App Teams

Practical lessons for developers, testers, and product managers drawn from app-store data, field testing, and the realities of mobile product quality.

View on Amazon

Available now

Build faster. Fool yourself less.

Testing AI is available on Amazon in print and Kindle editions.

Buy on Amazon Amazon
Front cover of Testing AI
Back cover of Testing AI
Front cover

Testing AI

Zoom in to inspect the cover, then flip to read the back. It is the same book, but the back cover gives the punchline: testing is becoming confidence engineering.

Buy on Amazon Amazon