Deployment and serving
Canary, blue-green and rolling releases; batch versus online serving; and the rollback that turns a bad version into a non-event.
A model that scores beautifully offline is still a hypothesis about production. Deployment is the experiment that tests it, and the whole design challenge is to run that experiment without exposing every user to it at once. What you want is a release in which a bad version is cheap, observable and reversible — a fraction of traffic, a clear signal, and a one-step rollback.
The simulator below puts a deliberately mediocre new version behind three strategies. Watch how canary limits exposure by ramping traffic, how blue-green switches everything at once, and how the rollback threshold decides between the two.
Choose a release strategy and a bad new version — watch the guardrail decide
bars = blended error · pale bars = % traffic on new version · dashed = threshold
Strategy
Rolled back at step 6 — blended error crossed 2.0%.
old-version error 1.0% · peak blended 2.01%
Canary limits exposure by ramping traffic, so a bad version trips the guardrail while only a fraction of users see it; blue-green switches everything at once and relies on a fast, clean rollback; rolling replaces instances in batches. The simulation assumes a single fixed error rate and instant rollback — real error rates are noisy and rollbacks take time.
Blast radius is a design choice
The same bad model can affect 1% of requests or 100%, for the same training cost, purely because of how it was released. Staged rollout is not a deployment convenience — it is the primary control on how much damage an inevitable mistake can do. Optimise for reversibility before you optimise for speed.
The three release patterns
- Canary — serve the new version to a small slice of traffic, watch its metrics, and ramp up only if they hold. Best error containment; needs traffic routing and enough volume for the slice's metrics to be meaningful.
- Blue-green — run two identical environments, switch traffic from blue to green atomically, and keep blue warm to switch back instantly. Clean rollback; the whole user base sees the switch at once.
- Rolling — replace instances in batches. No duplicate environment needed, but old and new versions coexist during the rollout, so the system must be compatible with both.
For models specifically, two extra patterns matter. Shadow deployment sends production input to the new model without showing anyone its output, so you can compare offline against live data safely. A/B testing splits traffic and measures the business outcome, which is the only way to know the model actually helped rather than merely scored well.
Serving: batch, online, and streaming
- Batch scoring runs on a schedule over stored data — cheapest, highest throughput, unacceptable latency for interactive use.
- Online serving answers per-request with the model resident in memory or behind an inference server; latency budgets of tens of milliseconds rule out heavy preprocessing at request time.
- Streaming scores events as they arrive, sitting between the two.
The recurring failure in all three is training-serving skew: the feature computed at training time differs from the one computed at serving time, often because the serving path cannot run the same expensive transformation. A feature store exists to make both paths read the same definition.
Careful
A model endpoint is not just a function. Inputs must be validated against the schema the model was trained on, because a silently renamed or unit-changed column produces confident garbage rather than an error. Version the model, the feature schema, and the code together; log every prediction with its inputs so incidents are interrogable; and make rollback a routine, one-command operation rather than an emergency procedure.
Cost, and the model you actually ship
Latency and cost are first-class constraints. A larger model that is two points more accurate may be unusable if it cannot meet the latency budget or its per-request cost dwarfs the value of the decision. Common levers are quantisation, distillation into a smaller student, caching, batching, and choosing a simpler model class when the accuracy gap does not justify the complexity. The best deployed model is the simplest one that clears the bar.
Illustrative vs real
The simulator uses one fixed error rate per version, instant rollback and a clean traffic split. Real error rates are noisy, rollback takes real time, and canary slices need enough volume to detect a difference. The traffic schedules and blended-error arithmetic are real; the environment is a simplification.
Check yourself
Eduspheria wiki · Data, MLOps & Deployment, MLOps
0 / 4 answered
From the assignment paper
Modeled on NITJ AI-511, Assignment/Quiz
0 / 6 answered
Where next: monitoring and drift — measuring, after release, whether the world has moved away from the model.