Section 006 · Chapter 1, The End of One-Run Testing
Determinism
Sometimes the right testing move is to turn down variation so the product, judge, or validation system becomes easier to reason about.
What to do
- Run the eval once with the most stable settings you can reasonably use: fixed model version if available, low temperature, fixed seed if supported, pinned retrieval data, stable tools, and frozen prompts.
- Define runnable checks that exercise determinism.
- Set acceptable outcomes and blocker failures for determinism before running the evaluation.
Evidence to preserve
- Preserve the inputs, versions, configurations, raw outcomes, and results for determinism needed to reproduce work on Determinism.
- Report results for determinism by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
Temperature, top_p, and top_k all affect sampling from the model's next-token probability distribution. They are usually applied after the model computes logits and before the next token is sampled. In many implementations, temperature rescales logits, top_p truncates by cumulative probability, and top_k truncates by rank. Different providers may apply these controls in different orders or expose only some of them.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "Determinism." Testing AI Knowledge Edition, section 6.
https://jarbon.ai/testing-ai/knowledge/ch006-determinism.html