Section 053 · Chapter 8, Operating AI: Observability, Relevance, and Economics
Synthetic Test Data
Synthetic data can expand coverage, but it can also manufacture a false picture of reality.
RAGsynthetic datacounterfactualsynthetic test data
What to do
- Use synthetic data to fill coverage gaps, not to replace reality.
- Ask for examples across languages, literacy levels, devices, regions, risk categories, and malformed inputs.
- Review them for realism, expected-answer quality, policy correctness, and whether they actually test the intended risk.
- Treat synthetic data as a hypothesis generator, not a substitute for measured production behavior.
Evidence to preserve
- Preserve the inputs, versions, configurations, raw outcomes, and results for RAG, synthetic data, counterfactual, synthetic test data needed to reproduce work on Synthetic Test Data.
- Report results for RAG, synthetic data, counterfactual, synthetic test data by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
Track synthetic-data provenance, generator model, prompt, seed, intended risk, reviewer approval, similarity to real data, and downstream failure discovery. Treat synthetic data as a hypothesis generator, not a substitute for measured production behavior.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "Synthetic Test Data." Testing AI Knowledge Edition, section 53.
https://jarbon.ai/testing-ai/knowledge/ch053-synthetic-test-data.html