Section 002 · Chapter 1, The End of One-Run Testing
What Makes a System Non-Deterministic?
Before builders can evaluate unpredictable systems, they need to understand where the unpredictability comes from and which variation actually matters.
What to do
- Ask the same model to summarize a document ten times and you may get ten different summaries.
- Log the weather snapshot, store inventory, delivery promise, driver state, user location, model route, tool outputs, and policy version.
- Define runnable checks that exercise determinism, personalization, and makes non deterministic.
Evidence to preserve
- Log the weather snapshot, store inventory, delivery promise, driver state, user location, model route, tool outputs, and policy version.
- Preserve the inputs, versions, configurations, raw outcomes, and results for determinism, personalization, makes non deterministic needed to reproduce work on What Makes a System Non-Deterministic?.
- Report results for determinism, personalization, makes non deterministic by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
Technically, builders should separate sources of randomness from sources of state and sources of platform change. Model temperature, random seeds, ranking tie-breakers, async timing, retrieval snapshots, feature flags, user profiles, tool outputs, dependency failures, provider model versions, safety filters, hardware/runtime paths, and hidden product context should be logged independently because each one creates a different debugging path. When a system has fallback paths, log which path produced the answer so a fluent but degraded response is not mistaken for a healthy one.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "What Makes a System Non-Deterministic?." Testing AI Knowledge Edition, section 2.
https://jarbon.ai/testing-ai/knowledge/ch002-makes-non-deterministic.html