Section 057 · Chapter 8, Operating AI: Observability, Relevance, and Economics
Cost and Token Budget Testing
AI quality includes whether the system can afford to behave that way.
latencyRAGtoken budgetcost token budget
What to do
- Track input tokens, output tokens, retrieved tokens, tool-call tokens, judge tokens, retry tokens, and total cost per task.
- Watch p95 and p99, not only averages.
- Measure quality per dollar by segment.
Evidence to preserve
- Track input tokens, output tokens, retrieved tokens, tool-call tokens, judge tokens, retry tokens, and total cost per task.
- Preserve the inputs, versions, configurations, raw outcomes, and results for latency, RAG, token budget, cost token budget needed to reproduce work on Cost and Token Budget Testing.
- Report results for latency, RAG, token budget, cost token budget by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
Cost testing should track token budgets by span, cache hit rate, retry count, tool-call count, model mix, latency percentiles, queue behavior, and marginal quality per dollar by task category.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "Cost and Token Budget Testing." Testing AI Knowledge Edition, section 57.
https://jarbon.ai/testing-ai/knowledge/ch057-cost-token-budget.html