The Wiki
Core6 min read

Convolution cost

Where the parameters and compute actually go in a conv layer — the arithmetic that steered a decade of architecture design.

Change a layer's shape and its bills move in quadratics you should feel intuitively, because every convnet design pitch ever pitched lives and dies on these two numbers:

params=k2⋅Cin⋅Cout+Cout \text{params} = k^2 \cdot C_{\text{in}} \cdot C_{\text{out}} + C_{\text{out}} compute=params×Wout×Hout \text{compute} = \text{params} \times W_{\text{out}} \times H_{\text{out}}

(Note the absence of input spatial size in the parameter count: a 7×7 kernel on a 224×224 image has exactly the same 147 weights as the same kernel on a 64×64 thumbnail. That's weight-sharing for you — and the exact reason dense layers are so much more expensive per-input at image scale.)

a single conv layer: change the knobs, watch the two bills move.

4.6K params

k²·C_in·C_out + C_out — unchanged by input size; that’s weight-sharing’s bill.

17.84M MACs

at output 62×62 (same weights stamped everywhere).

Dense-layer alternative doing the *equivalent* stamp (all positions, own weights): about 18.87M MACs — 1× the conv’s cost.

Note the trade you can't buy: kernel and channels enter the *parameter* count (k², C_in·C_out), the spatial size and stride only enter the *compute* count. Params decide memory; compute decides time.

Exact closed forms for one conv layer (same padding): params = k²·C_in·C_out + C_out; MACs = params·W_out·H_out. The dense comparison assumes fully-shared “same coverage” — an honest baseline that shows how much sharing buys at image scale.

The knobs trade against each other

What the artifact's dial covers cheaply:

  • Kernel size k enters squared (k2k^2), which is why the field standardized on 3×3 — going 5→3 with two stacked layers is cheaper and better.
  • Channel count enters twice (CinC_{\text{in}} and CoutC_{\text{out}}) — the real driver of cost. MobileNet-style architectures exist almost entirely on the insight that in × out coupling is the expensive term: do depth-wise spatial filtering and point-wise 1×11\times1 channel-mixing as separate cheap steps.
  • Output size enters the compute but not the parameters — so downsampling from pooling cuts the expensive part directly: same weights, s2s^2 fewer stamps.

Read this before you design anything

The ML-vision rulebook that emerged — mostly for arithmetic reasons:

  1. Downsample early and often; pay full resolution only where necessary.
  2. Channels grow as resolution shrinks (and the variable amount of per-layer compute stays roughly constant — see chapter 2's shrinking game).
  3. The bias term is trivial compared to k2CinCoutk^2 C_{\text{in}} C_{\text{out}}; don't waste effort optimizing it.

Illustrative vs real

Exact closed-form counts: params and multiply-accumulates for one conv layer, using the standard formulas with no padding effects (same-padding assumed for output size). Numbers here are layer-true; real model costs also fold in the activation/nonlinearity pass, normalization, and memory-bound overheads not counted in MACs.

Where next: transfer-learning — the punchline that follows: handcraft nothing, train nothing from scratch, borrow the bottom layers.

This lesson has exercises attached — costing layer configurations — launching once the exercises layer ships.