Wiki
Core11 min read

Why a GPU is not a fast CPU

A GPU wins by doing thousands of slow things at once, not one thing quickly. Peak throughput is arithmetic; reaching it is a question of regularity.

A datacentre GPU can sustain tens of trillions of floating-point operations per second. A high-end server CPU does a few trillion, while drawing less power. The obvious explanation — that GPU cores are individually faster — is wrong. A single GPU lane is slower and simpler than a CPU core: lower clock, no branch predictor worth the name, tiny cache. The GPU is fast for exactly one reason: there are thousands of them.

That single fact reorganises everything about how you program one. A CPU is engineered to finish one difficult instruction stream as fast as possible: speculation, out-of-order execution, deep caches. A GPU is engineered to keep thousands of simple lanes busy on work that looks the same. Optimising for a GPU is not "write the same code harder" — it is reshaping the problem until thousands of lanes have something identical to do.

Throughput, not latency

A CPU optimises latency — how long one task takes. A GPU optimises throughput — how many tasks finish per second. Neither is "better". Sending one small computation to a GPU is usually slower than the CPU, because you pay the transfer and setup before any parallel work starts. Parallelism is a bulk discount: worthless at quantity one.

A hundred dining philosophers, one waiter

Picture one brilliant chef who can complete any order in two minutes. Then picture a thousand line cooks who each take ten minutes per order. For one order the chef wins. For a banquet back-to-back, the line finishes far more plates per hour. GPUs are the line cooks.

Adjust the machine and the workload below. The filled bar is what the workload actually achieves; the dashed edge is the theoretical peak.

Size the machine and the workload — watch achieved throughput diverge

CPU323 GFLOP/s

16 cores × 8-wide SIMD × 1.8 GHz

GPU25.07 TFLOP/s

8,192 lanes × 1.8 GHz

Every lane does the same arithmetic on contiguous data — the shape GPUs were built for.

achieved speedup

78×

The dashed edge is each chip's theoretical peak; the filled bar is what this workload actually sustains. GPUs buy throughput with width, so they win big when every lane follows the same path and lose most of that edge when the work branches. Throughput is also not latency: a single narrow task is often still faster on the CPU. Numbers are illustrative.

The arithmetic of peak

Peak floating-point throughput is the product of how many multiply-accumulate units you have and how fast they run:

peak=Nlanes×fclock×2 FLOP/cycle\text{peak} = N_{\text{lanes}} \times f_{\text{clock}} \times 2 \ \text{FLOP/cycle}

The factor of two is because a fused multiply-add (FMA) counts as one multiply and one add. A chip with 8,192 lanes at 1.8 GHz therefore peaks around 8192×1.8×2≈29,0008192 \times 1.8 \times 2 \approx 29{,}000 GFLOP/s — 29 TFLOP/s. A 16-core CPU with 8-wide SIMD at the same clock peaks near 461 GFLOP/s. That is the raw gap: about 64×, and it comes entirely from lane count, not lane speed.

Peak is a ceiling, not a promise. Two limits bound a real kernel:

  • Compute bound — you have enough data in registers and the limit is arithmetic units. Matrix multiply wants to live here.
  • Memory bound — the arithmetic units idle waiting for data. Adding vectors is memory bound: one add per two loads, so bandwidth, not flops, is the wall.

The roofline model makes the boundary explicit. Define arithmetic intensity as flops per byte moved, I=FLOP/byteI = \text{FLOP}/\text{byte}. Then

attainable=min⁡ ⁣(peak, I×bandwidth)\text{attainable} = \min\!\left(\text{peak},\ I \times \text{bandwidth}\right)

Plot attainable performance against intensity and you get a roofline: a rising ramp (memory bound) that flattens into a ceiling (compute bound). The ridge point is where the two meet. GPUs have a high peak and a ridge point far to the right, which is the real reason they shine on dense linear algebra: matmul has enormous arithmetic intensity, so it climbs past the ramp and presses against the compute ceiling.

Why AI fits

Neural-network training and inference are, at the bottom, large dense matrix multiplications. Each output element is a dot product over a dimension typically in the hundreds or thousands, so the same weights are reused many times per byte fetched. That reuse is high arithmetic intensity, and high arithmetic intensity is the only thing that saturates a wide array. This is not a coincidence — it is why the GPU era and the deep-learning era happen to be the same era.

Parallelism maps onto the CPU/GPU tradeoff as well. Amdahl's law says the speedup from pp processors is bounded by the serial fraction ss:

S(p)=1s+1−spS(p) = \frac{1}{s + \dfrac{1-s}{p}}

As p→∞p \to \infty, S→1/sS \to 1/s. If 10% of your program cannot be parallelised, no amount of GPU width gets you past 10×. Widening the array only helps the part you actually parallelised.

Careful

"It has more cores" is not a performance argument. A kernel that branches heavily, walks a pointer graph, or touches memory irregularly starves the lanes and can run slower on a GPU than on a CPU — after you have also paid the host-to-device transfer. Measure with a profiler before rewriting, and remember that a small, latency-sensitive computation usually belongs on the CPU.

Illustrative vs real

The bars use round, order-of-magnitude numbers so the arithmetic is legible. They are not a benchmark of any named processor. The qualitative claims — throughput scales with lanes, memory-bound kernels stall well below peak, and irregular workloads lose most of the GPU advantage — hold on real hardware; the exact percentages will differ on every part and compiler.

Check yourself

Eduspheria wiki · Systems for AI, GPU computing

0 / 5 answered

  1. 1A GPU has 4096 lanes running at 1.5 GHz, each doing one FMA (2 FLOP) per cycle. What is its peak in GFLOP/s?
    GFLOP/s
    Numeric answer
  2. 2A kernel is memory bound. What does that mean?
    Multiple choice
  3. 3Sending a single small elementwise operation to a GPU is normally faster than running it on the CPU.
    True / false
  4. 4Which law bounds speedup by the fraction of a program that must run serially?
    Short answer
  5. 5A program is 80% parallelisable. What is the best possible speedup as the number of processors goes to infinity?
    ×
    Numeric answer

From the exam paper

Modeled on NITJ AI-619, End-Sem June 2025

0 / 5 answered

  1. 1A modern multi-core CPU organises its cache in several levels. Which ordering lists them from smallest and fastest to largest and slowest?
    Multiple choice
  2. 2A processor has three cache levels. If each level is 8 times larger than the one before it and the L1 is 32 KB, how large is the L3 in KB?
    KB
    Numeric answer
  3. 3A single GPU stream processor is individually faster than a CPU core at latency-sensitive, branch-heavy code.
    True / false
  4. 4Which statement best captures how a GPU's layout differs from both a modern CPU and a cluster of microprocessors?
    Multiple choice
  5. 5What is the term for a group of many microprocessors, each with its own memory, joined by a high-speed network to work on a single problem?
    Short answer

Where next: the CUDA execution model — the grid, block and thread hierarchy that turns a loop into lanes.