Testing AI

Second edition · By Jason Arbon

Testing AI

Engineering Confidence in Non-Deterministic Systems

The practical operating manual for people who have to decide whether AI-generated code, agents, models, and products are trustworthy enough to ship.

Buy on Amazon Available now Explore the Testing AI course

Second edition. Available now on Amazon as Kindle, paperback, and hardcover. Choose your format.

The second edition

Choose your format.

Testing AI second edition Kindle cover

Kindle

Read on Kindle, phone, or tablet.

Buy Kindle

Testing AI course · Video lessons coming soon

Turn uncertainty into evidence.

Learn to measure AI quality, compare judges, and make release decisions you can defend. Jason is preparing a video course based on Testing AI, with practical examples and exercises.

Explore the course

The book is now an AI agent

Put a Confidence Engineer beside your coding agent.

Testing AI can work inside Codex, Claude Code, Cursor, or Antigravity. Point it at the folder containing your code, prompts, test plans, requirements, logs, or documents. It maps risk, builds evidence, challenges the builder, and tells you whether the work should ship.

  • Risk map
  • Eval design
  • Independent review
  • Release evidence
  • Incident response
Your folder Code, docs, prompts, plans, logs, or requirements
Testing AI agent Independent Confidence Engineer
Decision-ready evidence
01System and risk map
02Focused checks and evals
03SHIP · CANARY · HOLD · ROLL BACK

Start here

No coding agent yet? Use Codex.

You do not need a repository or programming experience. Put whatever you want reviewed into one folder, then give Codex access to that folder.

Install Testing AI

Open Terminal, paste these two lines, then restart Codex. Existing installations are backed up automatically.

curl -fsSL https://testingaibook.com/install-testing-ai-codex.sh -o ~/Downloads/install-testing-ai-codex.sh
bash ~/Downloads/install-testing-ai-codex.sh
Download Codex plugin

Point it at your work

Choose the folder in Codex. Paste this prompt. The agent will inspect first and make uncertainty visible.

Use $testing-ai to inspect this folder as an independent Confidence Engineer. Map the system and its risks, run the smallest useful checks, and give me a scoped SHIP, CANARY, HOLD, or ROLL BACK recommendation. Show me the plan before changing anything.

Local: the plugin does not send telemetry or make hidden network calls. Controlled: it does not deploy, publish, merge, or approve releases for you.

Already use Claude Code, Cursor, or Antigravity?

Claude Code

Download the marketplace, unzip it, then add the marketplace and install testing-ai from Claude Code.

Download for Claude Code

Cursor

Download and unzip the native Cursor package, run its local installer, then reload the Cursor window.

Download for Cursor

Antigravity

Download and unzip the native Antigravity package, run its local installer, then restart Antigravity.

Download for Antigravity

The argument

Generation is easy. Validation is the hard part.

AI can produce more code, answers, designs, and decisions than any team can review the old way.

Most AI spending today still lives in experiments. The gap between an impressive demo and a production system is not usually another prompt trick. It is confidence: evidence that the system is useful, safe, economical, observable, reversible, and reliable enough for the people who depend on it.

Testing AI starts with testing because that is the doorway software teams recognize. It ends at confidence engineering: the operating discipline for shipping systems that vary from run to run, learn from changing data, call tools, personalize themselves, and behave differently under production load.

The book moves from first principles to operating practice: variance, sampling, statistical evidence, human and LLM judges, eval design, RAG, agents, generated code, security, observability, bias, interpretability, robotics, governance, and the future of AI testing itself.

Inside the book

A complete operating model for AI quality.

Not a catalog of tools. Not a stack of prompt recipes. The book connects measurement, engineering, operations, security, and judgment into one practical discipline.

Variance, sampling, and uncertainty

Replace one-run demos with repeated measurements, slices, confidence intervals, rare-failure hunting, and release evidence.

Statistics that support decisions

Use p-values, effect sizes, power, paired tests, calibration, abstention, and practical significance without statistical theater.

Judges, raters, and evals

Design rubrics, calibrate humans and LLM judges, measure disagreement, manage labels, and build evals tied to product risk.

RAG, agents, and generated code

Test retrieval separately from generation, score tool trajectories, verify integration behavior, and catch code that looks right but is wrong.

Security, safety, and containment

Threat-model prompt injection, MCP permissions, poisoning, guardrails, dangerous capabilities, deception, and containment boundaries.

Operations and the future

Build observability, SLOs, canaries, incident response, cost controls, robotics simulation, governance, and continuous AI-led testing.

All 21 chapters

Part I

The New Shape of AI Quality

  1. 1The End of One-Run Testing
  2. 2From Tests to Release Evidence
  3. 3Sampling and Uncertainty
  4. 4Statistical Tests for AI Quality
  5. 5Judges, Humans, and Disagreement
Part II

Evidence, Evals, and Production Readiness

  1. 6Building Evals That Matter
  2. 7Release Readiness for AI Systems
  3. 8Operating AI: Observability, Relevance, and Economics
Part III

AI-Generated Software and Confidence Engineering

  1. 9Generated Code Changes the Job
  2. 10Anti-Patterns That Create False Confidence
  3. 11The Confidence Engineer
Part IV

Data, Security, Safety, and Model Internals

  1. 12Data, Bias, Raters, and Incentives
  2. 13AI Security and Guardrails
  3. 14Frontier Safety and Containment
  4. 15How Models Work
  5. 16Introspection: White-Box Testing Networks
Part V

Future Systems and the Practical Playbook

  1. 17Personalized and Dynamic AI Products
  2. 18Embodied and Long-Running AI Systems
  3. 19Governance, Regulation, and Moral Futures
  4. 20The Practical Playbook
  5. 21Predictions for the Tokenized Product Future

Every chapter is also available as a short, mobile-first PDF carousel. Read it in your browser or download it to share on LinkedIn.

01
The End of One-Run TestingOpen PDF carousel
 02
From Tests to Release EvidenceOpen PDF carousel
 03
Sampling and UncertaintyOpen PDF carousel
 04
Statistical Tests for AI QualityOpen PDF carousel
 05
Judges, Humans, and DisagreementOpen PDF carousel
 06
Building Evals That MatterOpen PDF carousel
 07
Release Readiness for AI SystemsOpen PDF carousel
 08
Operating AIOpen PDF carousel
 09
Generated Code Changes the JobOpen PDF carousel
 10
Anti-Patterns That Create False ConfidenceOpen PDF carousel
 11
The Confidence EngineerOpen PDF carousel
 12
Data, Bias, Raters, and IncentivesOpen PDF carousel
 13
AI Security and GuardrailsOpen PDF carousel
 14
Frontier Safety and ContainmentOpen PDF carousel
 15
How Models WorkOpen PDF carousel
 16
Introspection: White-Box Testing NetworksOpen PDF carousel
 17
Personalized and Dynamic AI ProductsOpen PDF carousel
 18
Embodied and Long-Running AI SystemsOpen PDF carousel
 19
Governance, Regulation, and Moral FuturesOpen PDF carousel
 20
The Practical PlaybookOpen PDF carousel
 21
Predictions for the Tokenized Product FutureOpen PDF carousel

Who it is for

For everyone now responsible for confidence.

AI is collapsing the old walls between building, testing, product judgment, and operating software. The readers differ; the release decision is shared.

Developers and coding-agent users

Learn how to validate code and systems you did not fully write, including hidden integration, security, data, and operational failures.

Testers and automation engineers

Move beyond brittle exact assertions into evals, probabilistic evidence, production traces, risk slices, and human-machine review systems.

AI product builders and architects

Connect prompts, models, data, retrieval, tools, policies, user experience, cost, and release controls into one testable system.

Engineering leaders and executives

Ask for evidence that supports ship, hold, canary, rollback, investment, governance, and customer-risk decisions.

What changes after reading

You stop asking whether AI “works.”

You start asking the sharper questions that turn ambiguous behavior into engineering evidence.

  1. Which kinds of variation are harmless, and which are product failures?
  2. What population did we sample, and which risks are hidden by the average?
  3. Who or what is judging quality, and has that measurement system been calibrated?
  4. Did the system retrieve the right evidence, use tools safely, and preserve an auditable trace?
  5. Are quality gains worth their latency, compute cost, operational complexity, and user impact?
  6. What evidence justifies release, and what exact signal should trigger rollback?

Draft reviewers

Thank you to the early readers who made the book better.

This book improved because people took time to read rough chapters, mark confusing passages, suggest missing topics, and push the material toward practical usefulness for the broader AI quality community.

Karime Salomon Zarate

Missing topics, definitions, chatbot quality criteria, invisible dependency failures, editorial clarity, and practical reviewer perspective.

Jeffery Evans

Regulatory stakes, ISO/IEC 42001 context, beginner-friendly SKILL.md explanation, reader navigation, sampling definitions, p-value clarity, and release-gate examples.

Gastón Marichal

Non-deterministic framing, varying degrees of correctness and failure, and clearer calibration language.

Christopher L.

Editorial clarity around confidence-interval language and a more beginner-friendly explanation of how intervals are calculated.

Richard Hill

Quality dimensions, sample-size nuance, rare-failure hunting, AI judge bias questions, and practical evaluator/tooling suggestions.

Andre Hulse

Front-matter positioning, preface structure, reader framing, and a clearer bridge from traditional software quality into AI quality.

Steven Knopf

Ask AI link behavior, dense-sentence cleanup, link hygiene, repeated-example spotting, and draft-reader experience.

Kiran Rayachoti

The suggestion to treat performance engineering as a named AI quality discipline rather than a narrow testing activity.

Sergio Segura

LLM-as-a-judge cost realism, practical eval tooling, and multi-judge voting nuance for high-stakes evaluation.

Zoltán Tarkó

Senior-engineering feedback on tone, accountable AI builders, pass/fail nuance, judge confidence wording, and LLM-judge consistency.

Larry Petersen

Terminology consistency around testers, quality engineers, and Confidence Engineers.

Jason Arbon
Jason Arbon
Founder · AI quality builder · author
Connect with Jason on LinkedIn ↗

Also by Jason

How AI Tests Software cover How AI Tests SoftwareAmazon ↗ How Google Tests Software cover How Google Tests SoftwareAmazon ↗ App Quality cover App QualityAmazon ↗

About the author

Jason has spent his career ensuring the quality of large, complex, and non-deterministic systems.

Jason Arbon is an entrepreneur, co-author of How Google Tests Software, and founder of AI testing companies including test.ai and Testers.ai. He has built and led quality systems across Microsoft, Bing, Google Search, Chrome, ChromeOS, Applause, and AI-first testing products.

His work sits at the intersection of software engineering, product judgment, statistics, human evaluation, and autonomous testing. Testing AI brings those experiences together into Confidence Engineering: the discipline of building justified confidence in software created by and with AI.

LinkedIn Jason's projects Testers.ai Writing

Also by Jason Arbon

Two earlier books about quality at scale.

The new book extends a long-running argument: quality becomes useful when teams turn it into a system, not a final inspection.

Cover of How Google Tests Software
Published

How Google Tests Software

Co-authored with James Whittaker and Jeff Carollo. An inside account of testing roles, infrastructure, and engineering culture at Google scale.

View on Amazon
Cover of App Quality: Secrets for Agile App Teams
Published

App Quality: Secrets for Agile App Teams

Practical lessons for developers, testers, and product managers drawn from app-store data, field testing, and the realities of mobile product quality.

View on Amazon

Available now

Build faster. Fool yourself less.

Testing AI is available on Amazon in print and Kindle editions.

Buy on Amazon Amazon
Front cover of Testing AI
Back cover of Testing AI
Front cover

Testing AI

Zoom in to inspect the cover, then flip to read the back. It is the same book, but the back cover gives the punchline: testing is becoming confidence engineering.

Buy on Amazon Amazon