Wiki
Core6 min read

Convolution cost

Where the parameters and compute actually go in a conv layer — the arithmetic that steered a decade of architecture design.

Change a layer's shape and its bills move in quadratics you should feel intuitively, because every convnet design pitch ever pitched lives and dies on these two numbers:

params=k2⋅Cin⋅Cout+Cout \text{params} = k^2 \cdot C_{\text{in}} \cdot C_{\text{out}} + C_{\text{out}} compute=params×Wout×Hout \text{compute} = \text{params} \times W_{\text{out}} \times H_{\text{out}}

(Note the absence of input spatial size in the parameter count: a 7×7 kernel on a 224×224 image has exactly the same 147 weights as the same kernel on a 64×64 thumbnail. That's weight-sharing for you — and the exact reason dense layers are so much more expensive per-input at image scale.)

a single conv layer: change the knobs, watch the two bills move.

4.6K params

k²·C_in·C_out + C_out — unchanged by input size; that’s weight-sharing’s bill.

17.84M MACs

at output 62×62 (same weights stamped everywhere).

Dense-layer alternative doing the *equivalent* stamp (all positions, own weights): about 18.87M MACs — 1× the conv’s cost.

Note the trade you can't buy: kernel and channels enter the *parameter* count (k², C_in·C_out), the spatial size and stride only enter the *compute* count. Params decide memory; compute decides time.

Exact closed forms for one conv layer (same padding): params = k²·C_in·C_out + C_out; MACs = params·W_out·H_out. The dense comparison assumes fully-shared “same coverage” — an honest baseline that shows how much sharing buys at image scale.

The knobs trade against each other

What the artifact's dial covers cheaply:

  • Kernel size k enters squared (k2k^2), which is why the field standardized on 3×3 — going 5→3 with two stacked layers is cheaper and better.
  • Channel count enters twice (CinC_{\text{in}} and CoutC_{\text{out}}) — the real driver of cost. MobileNet-style architectures exist almost entirely on the insight that in × out coupling is the expensive term: do depth-wise spatial filtering and point-wise 1×11\times1 channel-mixing as separate cheap steps.
  • Output size enters the compute but not the parameters — so downsampling from pooling cuts the expensive part directly: same weights, s2s^2 fewer stamps.

Read this before you design anything

The ML-vision rulebook that emerged — mostly for arithmetic reasons:

  1. Downsample early and often; pay full resolution only where necessary.
  2. Channels grow as resolution shrinks (and the variable amount of per-layer compute stays roughly constant — see chapter 2's shrinking game).
  3. The bias term is trivial compared to k2CinCoutk^2 C_{\text{in}} C_{\text{out}}; don't waste effort optimizing it.

Illustrative vs real

Exact closed-form counts: params and multiply-accumulates for one conv layer, using the standard formulas with no padding effects (same-padding assumed for output size). Numbers here are layer-true; real model costs also fold in the activation/nonlinearity pass, normalization, and memory-bound overheads not counted in MACs.

Check yourself

Eduspheria wiki · Deep Learning, Convolutions and vision

0 / 5 answered

  1. 1A conv layer uses 3 by 3 kernels with 64 input channels, 128 output channels and a bias per filter. Using params equal to k squared times C_in times C_out plus C_out, how many parameters does it have?
    parameters
    Numeric answer
  2. 2Why did the field standardise on 3 by 3 kernels?
    Multiple choice
  3. 3The parameter count of a convolutional layer depends on the input image's spatial size.
    True / false
  4. 4Which quantity drives conv cost because it appears twice in the parameter formula?
    Short answer
  5. 5For the layer above (73,728 weights, output 32 by 32, bias ignored), compute equals params times output width times output height. How many multiply-accumulates does it cost, rounded to the nearest million?
    million MACs
    Numeric answer

Where next: transfer-learning — the punchline that follows: handcraft nothing, train nothing from scratch, borrow the bottom layers.

This lesson has exercises attached — costing layer configurations — launching once the exercises layer ships.