Serving and latency: the batching tradeoff

Here is the collision at the heart of inference serving. A GPU is a throughput machine, so it wants large batches: one big matrix multiply keeps thousands of lanes busy and amortises the fixed overhead of a kernel launch. But your users are a latency machine: each request wants its answer now, and a large batch means each request waits while the batch fills. Push throughput up and latency degrades; shrink the batch for latency and the GPU starves. The entire job of a serving system is to sit at a good point on that curve — or, better, to change the curve.

The tool you reach for first is batching, because it is simple and effective. Then you discover its limits, and the modern systems — continuous batching, paged attention, speculative decoding — are all attempts to keep the throughput of large batches without paying large-batch latency.

Move the batch size and watch latency and throughput move in opposite directions.

The latency budget #

End-to-end latency is a sum of stages, and reasoning about it means naming each one:

Batching acts on the middle two. Preprocessing overhead is roughly fixed per call, so it is amortised across the batch; compute scales with batch size but sub-linearly on a GPU, because the same weights serve the whole batch. That sub-linear compute is the only reason batching wins, and it is the reason the throughput curve in the widget flattens instead of running away.

Queueing is where the latency actually comes from under load. Little's law relates the three quantities you cannot independently choose:

where $L$ is the number of requests in the system, $\lambda$ the arrival rate, and $W$ the average time each spends there. If arrivals exceed service rate, $L$ grows without bound and so does latency — the queue is unstable. This is why "just add a bigger batch" fails: past the throughput ceiling, extra load becomes queueing delay, not completed work.

The batching arithmetic #

For a batch of size $B$, with fixed per-call overhead $o$ and per-item compute $t$:

Throughput rises with $B$ and saturates at $1/t$ (the overhead vanishes as a fraction). Waiting time to fill a batch grows roughly linearly in $B$ — under Poisson arrivals the mean wait is about $(B-1)/(2\lambda)$ — so latency grows while throughput stops improving. The useful region is the knee, not the asymptote. This is textbook queueing; there is no clever trick that makes latency and throughput improve together for a fixed model and hardware.

Changing the curve: continuous batching #

Static batching is wasteful for generative models because requests do not finish together. A batch is limited by its longest sequence; short requests sit idle occupying memory while the long one runs. Continuous batching (iteration-level scheduling, as in Orca and vLLM) fixes this by rebuilding the batch every decoding step: finished sequences leave, queued ones join, and no slot is wasted. Paged attention attacks the memory side, storing the KV cache in non-contiguous pages so sequences of different lengths can share GPU memory without fragmentation. Together they move the curve: higher throughput at the same latency, because the batch no longer waits for its slowest member.

The same idea generalises. Speculative decoding uses a small draft model to propose tokens a large model verifies in parallel, converting idle capacity into lower latency. Every technique here is a way of buying back the gap between the hardware's ideal batch and the user's impatience.

The tail is what users feel #

Mean latency is a comfortable lie. Users experience the slow requests, so serving is managed on percentiles — p50, p95, p99, p99.9. At scale the tail dominates experience and, worse, it amplifies. If one page load fans out to 100 backend calls and each has a 1% chance of being slow, the probability that at least one is slow is $1 - 0.99^{100} \approx 63\%$ — the page is slow most of the time even though every service looks healthy. The standard defences are hedged requests (send a backup after the p95 and take the winner), tied requests (cancellation so wasted work stops), and simply keeping fan-out low.

SLOs and goodput #

An SLO is the contract that turns these numbers into engineering. Goodput is the rate of requests that actually meet the SLO — work that arrives too late is, for the user, work not done. Capacity planning inverts the question: rather than maximising throughput, find the load at which p99 crosses the SLO and provision headroom below it. The load-shedding question ("what do we drop first when the queue grows?") belongs in the design, decided in advance, not at 3 a.m.

Where next: with the serving path in hand, the natural continuation is the training side of the same stack — data pipelines and distributed training — the systems that produce the checkpoints this chapter deploys.