Section 044 · Chapter 6, Building Evals That Matter
The Asymptotic Curve of AI Quality
AI quality usually improves quickly at first, then gets harder, slower, and never reaches perfection.
What to do
- Define runnable checks that exercise rubric, asymptotic curve, and retrieval.
- Set acceptable outcomes and blocker failures for rubric, asymptotic curve, and retrieval before running the evaluation.
- Run representative cases for rubric, asymptotic curve, and retrieval and preserve the failures that would change the decision.
Evidence to preserve
- Preserve the inputs, versions, configurations, raw outcomes, and results for rubric, asymptotic curve, retrieval, asymptotic curve quality needed to reproduce work on The Asymptotic Curve of AI Quality.
- Report results for rubric, asymptotic curve, retrieval, asymptotic curve quality by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
Plot quality over time with uncertainty bands, not just point estimates. Look for diminishing returns, plateaus, slice-specific ceilings, and architecture-limited performance. The asymptote is not an excuse to stop testing. It is evidence that the next improvement may require a different model, data source, workflow, containment layer, or product boundary.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "The Asymptotic Curve of AI Quality." Testing AI Knowledge Edition, section 44.
https://jarbon.ai/testing-ai/knowledge/ch044-asymptotic-curve-quality.html