From handcrafted to learned features

For decades, if you wanted a machine to recognise a pedestrian, match a landmark between two photographs, or find a face, you wrote down by hand what a feature should look like. SIFT computed histograms of gradient orientations around keypoints; HOG computed the same histograms on a dense grid; Haar features compared rectangular patches of brightness; LBP encoded each pixel as a binary comparison with its neighbours. These descriptors were not learned. They were engineered — designed, tuned and defended in papers.

Then in 2012 a deep network called AlexNet cut the ImageNet error rate by a wide margin, and within a few years almost every step those descriptors performed by hand was being learned from data instead. The surprising part was not that learning won on a benchmark. It was how much of the hand-engineering turned out to be unnecessary.

The widget below puts the two pipelines side by side on the same patch. In the handcrafted view you watch gradients become per-cell orientation histograms and then one normalized vector; in the learned view four kernels convolve the patch into feature maps and a pooled embedding. Translate the patch and compare how each representation holds up.

What handcrafted features encode #

The classical descriptors differ in detail but share one move: pick a simple local measurement, aggregate it over a region, and normalize so the result survives the changes you do not care about.

• SIFT measures gradient orientation at scale-space keypoints, bins the orientations in a 4×4 neighbourhood into a 128-dimensional vector, and subtracts the dominant orientation so the descriptor rotates with the patch.
• HOG drops the keypoint detector and computes the same kind of histogram on a dense grid of cells, contrast-normalizing overlapping blocks; it was the standard pedestrian detector for a decade.
• Haar features are differences of rectangular sums — cheap edge and centre-surround measurements computed instantly with an integral image — and a boosted cascade of them powered real-time face detection.
• LBP compares each pixel with its neighbours and emits a short binary code, an illumination-robust texture measure.

The common formula is a magnitude-weighted orientation histogram. For cell $c$ and orientation bin $b$,

$$h_c[b] = \sum_{p \in c} \lVert \nabla I(p) \rVert \, \mathbb{1}\!\left[\theta(p) \in \text{bin } b\right],$$

followed by a normalization (L2, or block contrast normalization in HOG) that removes overall brightness changes. Every choice here — cell size, bin count, which normalizer — is a design decision made by a person.

What changed: learn the filters #

A convolution layer is the same operation a handcrafted filter performs, with the numbers left free. For an input image $I$ and filters $W_k$,

$$F_k(x,y) = \sigma\!\left(\sum_{i,j} W_k(i,j)\, I(x+i,\, y+j) + b_k \right).$$

In SIFT or HOG the kernel is a fixed gradient operator and the weights inside it are chosen by the author. In a CNN the $W_k$ are parameters initialised at random and updated by backpropagation to reduce a task loss. Stack several layers and each layer composes the previous one's features, so the network can build measurements that no one wrote down.

AlexNet trained roughly 60 million parameters over 1.2 million labelled images and 1000 classes, and its learned filters beat every handcrafted pipeline on the benchmark. The lesson generalised: given enough data and compute, the measurement itself becomes a learned parameter.

What a CNN's filters look like at depth #

The filters are not arbitrary. When you visualise what a trained image network has learned, a consistent hierarchy appears:

• First layers resemble oriented edge and colour-blob detectors — Gabor-like filters that look strikingly like a learned SIFT or HOG front end.
• Middle layers combine edges into corners, junctions and simple textures.
• Deeper layers respond to object parts and motifs: wheels, eyes, windows.
• Late layers become class-selective, firing for whole objects and categories.

Nobody specified that hierarchy; it falls out of composing trainable filters and training them end to end. The handcrafted descriptors spent years trying to capture the first two levels by hand; the network learned all four.

Transfer learning: a pretrained CNN as a descriptor #

The most practical bridge from classical to learned features is also the simplest. Take a CNN trained on a large dataset, remove its classification head, and use the activations it produces as your descriptor:

1. Run the input through the frozen convolutional stack.
2. Read a feature map (per location) or globally pool it into an embedding (per image).
3. L2-normalize and compare with cosine similarity or a nearest-neighbour search — exactly as you would compare SIFT vectors.

Razavian and colleagues showed in 2014 that these off-the-shelf ImageNet features, used without any fine-tuning, already rivalled or beat handcrafted pipelines across many recognition tasks. That is the operational meaning of "a CNN is a drop-in replacement for SIFT": the frozen network is a fixed function from pixels to a vector, and the vector behaves like a descriptor. If your task is close to the pretraining data, freezing is enough; if there is a large domain gap and enough data, fine-tuning adapts the filters too.

When classical still wins #

Learned features dominate large-scale natural-image recognition, but the classical toolbox has not gone away. It is still the right choice when:

• Data is scarce or the modality is unusual. No large pretraining corpus exists, and a handcrafted descriptor needs no labels to define.
• Compute, latency or power is tight. A gradient histogram or Haar sum is a handful of operations per pixel, with no GPU and no model weights to ship.
• You need interpretability or guarantees. Each dimension of a SIFT/HOG vector has a stated meaning; a regulated system may require features a human can audit rather than a distributed code.
• The task is geometry, not semantics. Camera calibration, epipolar geometry, absolute pose, photogrammetry and structure-from-motion are solved by explicit geometric algorithms. Classical detectors and descriptors with RANSAC remain robust and deterministic, and matching precision at the sub-pixel level still favours them.

In practice the two combine: a learned detector and descriptor propose matches, and classical geometric verification decides which are consistent.

Descriptors and learned embeddings #

Strip away the terminology and both objects are the same thing: a vector in a space where distance means similarity. Matching is a nearest-neighbour query in both cases, and cosine or Euclidean distance is the metric either way.

The difference is where the invariance lives. In a handcrafted descriptor, invariance to rotation or illumination is inserted by a person, step by step. In a learned embedding, invariance is whatever the training objective forced the representation to have — contrastive or triplet losses pull similar inputs together and push others apart, so the geometry of the space is learned, not specified. The bag-of-visual-words histogram is the historical halfway house: a learned vocabulary over handcrafted descriptors. A modern embedding is its end-to-end descendant.

Check yourself

Eduspheria wiki · Applied AI, Classical computer vision

0 / 6 answered

1. 1A SIFT-style descriptor uses a 4×4 grid of spatial cells with an 8-bin orientation histogram per cell. How many dimensions does the descriptor have?

   Check Hint Show answerNumeric answer
2. 2When the filters of the first convolution layer of a CNN trained on natural images are visualised, they most resemble…

   A.Random noise patternsB.Per-pixel class histogramsC.Semantic object detectorsD.Oriented edge and colour-blob detectors

   Check Show answerMultiple choice
3. 3Global average pooling of a convolutional feature map discards where activations occurred, which gives the resulting embedding translation invariance.

   TrueFalse

   Check Show answerTrue / false
4. 4Which 2012 network, trained on 1.2 million ImageNet images, showed that learned convolutional filters decisively beat handcrafted pipelines?

   Check Show answerShort answer
5. 5An image patch has rows [[0, 0, 10], [0, 0, 10], [0, 0, 10]]. Applying the vertical Sobel kernel [[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]], what is the centre-pixel response?

   Check Hint Show answerNumeric answer
6. 6For which task would a classical handcrafted pipeline still commonly be preferred over a pretrained CNN?

   A.Detecting objects in a dataset almost identical to ImageNetB.Captioning arbitrary natural imagesC.Sub-pixel camera pose estimation from calibrated multi-view geometryD.Classifying a large set of consumer photographs

   Check Show answerMultiple choice

Where next: the same classical-to-learned story plays out in language — we start with the fixed statistics of n-gram language models before moving to learned representations.