Cloud-native AI: containers and orchestration
A trained model is not a service. Containers freeze the environment, an orchestrator reconciles a declared replica count, and the loop keeps serving through failure.
A trained model is a file, not a service. To make it useful it has to load weights, accept requests, run inference, handle many concurrent callers, and survive a machine dying at 3 a.m. Most of that is not machine learning at all — it is the ordinary problem of running software reliably, and it has an established answer: package it in a container and let an orchestrator run many copies of it.
The container solves "works on my machine" by freezing the environment — libraries, CUDA version, drivers, entrypoint — into one immutable image. The orchestrator solves "what if a replica dies" by treating your stated intent as the source of truth and continuously making reality match. Together they are what data scientists mean when they say a model was "productionised".
Declarative, not imperative
With a script you say how: start this, then that, and if it dies you notice. With an orchestrator you say what: three replicas of this image, always. The controller loop compares desired state to observed state and acts on the difference. Failure becomes a normal input, not an exception.
Provision replicas, kill a pod, and watch the capacity and reconciliation status change.
Declare a replica count, kill a pod, watch reconciliation and saturation
node 0
node 1
capacity
600 req/s
load
500 req/s
utilisation
83%
Kubernetes, Mesos and Borg all share the reconciliation loop: you write the desired state, a controller drives observed toward it. Two consequences matter for AI serving — replicas give you both capacity and a failure domain, so losing one pod costs a fraction of throughput, and autoscaling is just a controller watching a metric like utilisation and editing the desired replica count. Sizes and rates are illustrative.
The container as a unit of deployment
A container image bundles the application with its user-space dependencies and runs as an isolated process on a shared kernel. For AI serving the image is large — a framework, a runtime, often multi-gigabyte CUDA libraries — but it is reproducible. The same digest runs on your laptop, a staging node, and the GPU pool, which removes an entire class of "it worked yesterday" failures.
Two consequences shape how you build these images:
- Layers and caching. Weights and dependencies change at different rates; order the image so the slow, rarely-changed layers (CUDA, Python packages) come first and application code comes last.
- Startup time. Orchestrators must start and stop replicas, so time to first inference matters — loading a multi-gigabyte model on every cold start is a real availability cost.
The reconciliation loop
The core abstraction is a control loop. You submit a Deployment saying "four replicas of image X". The orchestrator schedules four Pods onto Nodes, subject to resource requests (CPU, memory, and for AI, GPUs). Then it does nothing but watch. If a pod exits or a node fails, the observed count drops below four, and the controller schedules a replacement. There is no "restart" command in the normal flow; you never tell it what went wrong, because it does not care why — only that desired and observed differ.
This inversion is why Kubernetes-style systems won: every failure mode (crash, node loss, bad deploy, scale event) becomes the same diff-and-act operation, and the controller only ever has to be correct about the desired state.
Capacity, scaling and failure
Replicas buy two things at once. The first is capacity: total throughput is roughly replicas times per-replica throughput, until a shared dependency saturates. The second is a failure domain: losing one of four replicas costs a quarter of capacity, not the service. The orchestrator restarts the missing pod, and a load balancer stops routing to it in the meantime.
Horizontal scaling (more replicas) is the cloud-native default because it is boring and safe. Vertical scaling (a bigger GPU per replica) is sometimes necessary — a 70-billion-parameter model may not fit on one card — but it changes the failure domain: one big replica is now a single point of failure. Autoscaling is just a controller watching a metric (queue depth, GPU utilisation, requests per second) and editing the desired replica count, which is why the metric you choose is a design decision, not a detail.
GPU nodes and scheduling
GPUs make scheduling harder. They are scarce, they cannot be shared as fluidly as CPU time, and a pod either fits on a node's GPU memory or it does not. Orchestrators handle this with node pools (GPU nodes versus CPU nodes), resource requests and limits, and increasingly with time-slicing or MIG partitioning so several small models share one physical device. The practical rule: schedule a GPU the way you would schedule any other exclusive, expensive resource, and make the scheduler's decision explicit rather than relying on luck.
Careful
Containers isolate but do not secure by default. A container shares the host kernel, and the wrong flags — privileged mode, a mounted Docker socket, host networking — turn isolation into theatre. For AI specifically, model weights and tokens are often baked into images or environment variables, where they are easy to leak; prefer a secrets store mounted at runtime and scan images before they reach a registry. And an autoscaler without a cap is a bill, not a feature.
Illustrative vs real
The topology uses one load balancer, uniform pods and an instant restart, so the reconciliation idea is legible. Real clusters add scheduling delays, readiness and liveness probes, termination grace periods, pod disruption budgets and image-pull latency — the time from "pod died" to "capacity restored" is seconds to minutes, not instant. Per-replica request rates are round numbers, not a benchmark.
Check yourself
Eduspheria wiki · Systems for AI, Serving at scale
0 / 5 answered
From the exam paper
Modeled on NITJ AI-619, End-Sem June 2025
0 / 5 answered
Where next: the data layer of an AI service — vector databases and the approximate indexes that make similarity search fast.