Section 056 · Chapter 8, Operating AI: Observability, Relevance, and Economics
Canary, Shadow, and Rollback Strategy
Non-deterministic systems should earn traffic gradually, with clear rollback rules.
latencyescalationcanaryshadow moderollbackcanary shadow rollback strategy
What to do
- Start with low-risk categories when possible, watch the canary separately from the rest of production, then expand by segment only when the evidence stays clean.
- Do not decide after seeing a bad result whether it was bad enough to count.
- Do not promote because one run looked good.
- Choose one primary outcome before looking at results, then add guardrail metrics.
- Do not repeatedly peek at an ordinary fixed-horizon p-value and stop the first time it crosses 0.05.
Evidence to preserve
- Record model, prompt, policy, retrieval, and routing changes during the experiment; silently changing the treatment halfway through makes the result hard to interpret.
- Preserve the inputs, versions, configurations, raw outcomes, and results for latency, escalation, canary, shadow mode needed to reproduce work on Canary, Shadow, and Rollback Strategy.
- Report results for latency, escalation, canary, shadow mode by relevant slice, separate blocker failures from averages, state uncertainty and blind spots, and connect the result to a release decision.
Expert note
The deeper move is to treat rollout as a measured system. Define exposure units, segment gates, guardrail metrics, rollback thresholds, statistical confidence requirements, monitoring windows, human review queues, and post-release trace mining.
Continue the conversation
Apply this to your context.
Save your product context once, then open a focused conversation that combines it with this concept.
Cite this page
Jason Arbon. "Canary, Shadow, and Rollback Strategy." Testing AI Knowledge Edition, section 56.
https://jarbon.ai/testing-ai/knowledge/ch056-canary-shadow-rollback-strategy.html