Wiki
Core14 min read

Object detection and tracking

From scored boxes to real object trajectories: IoU, non-maximum suppression, mean average precision, and tracking-by-detection with Kalman filters and data association.

A detector does not see one object; it sees thousands of candidates. Every possible location and size is scored, and most scores are low. The engineering that turns that noisy cloud of boxes into a short, clean list — and then links the list across frames into trajectories — is the subject of this lesson.

The two quantities you need before anything else are intersection over union (IoU), which measures how much two boxes overlap, and non-maximum suppression (NMS), which uses IoU to delete duplicates.

Detection then deduplication

A detector scores many overlapping hypotheses per object. Sort them by confidence, keep the best, delete every remaining box that overlaps it too much, and repeat. That greedy loop is NMS — simple, and still the default post-processing in most deployed detectors.

Drag the IoU threshold and toggle NMS to see duplicates appear and vanish.

A detector emits many overlapping boxes per object. Greedy NMS keeps the highest-scoring box and suppresses every neighbour whose IoU exceeds the threshold.

car 0.99sign 0.99person 0.97car 0.91person 0.89sign 0.84person 0.78sign 0.72car 0.71person 0.63
candidates
10
kept
5
suppressed
5
hovered vs top IoU
—

Raise the threshold toward 1.0 and near-duplicates survive — lower it and each object collapses to one box.

Real IoU and greedy suppression on illustrative boxes — no actual neural detector or image is present. Class-aware NMS, Soft-NMS and learned set prediction (DETR) all generalise this same duplicate-removal idea.

IoU and non-maximum suppression

For boxes AA and BB, with intersection area ∣A∩B∣|A \cap B| and union ∣A∪B∣|A \cup B|:

IoU⁡(A,B)=∣A∩B∣∣A∪B∣.\operatorname{IoU}(A, B) = \frac{|A \cap B|}{|A \cup B|}.

NMS proceeds greedily: sort boxes by score; take the highest; remove every box whose IoU with it exceeds a threshold τ\tau; repeat on what remains. The threshold is a genuine knob — too low and nearby distinct objects are merged, too high and duplicates survive.

Measuring a detector: precision, recall and mAP

Detection is graded with the same confusion-matrix ideas as classification, but "correct" now depends on a spatial criterion. A detection is a true positive if its IoU with a ground-truth box of the same class exceeds a threshold (0.5 by convention). Sweeping the confidence threshold traces a precision–recall curve, and the area under it — average precision — averaged over classes gives mAP. Modern benchmarks report mAP at several IoU thresholds (e.g. COCO's mAP@[.5:.95]).

Tracking by detection

Given boxes per frame, tracking asks which box in frame tt is the same object as a box in frame t−1t-1. A standard recipe is SORT: represent each track with a Kalman filter over position, size and velocity; predict each track forward; solve the assignment between predictions and detections by IoU using the Hungarian algorithm; update matched tracks, start new ones, and drop tracks unmatched for too long.

Identity is not detection

A detector localises; only tracking carries identity. Two box sequences can be locally perfect and still swap identities the moment they cross, occlude, or leave and re-enter the frame. Identity switches are the failure mode that detection metrics do not capture and tracking metrics (MOTA, IDF1) do.

Illustrative vs real

The widget uses hand-placed candidate boxes and pure IoU arithmetic — there is no image and no neural network. Real detectors regress boxes from learned features (anchors, or anchor-free centres), and modern trackers add appearance embeddings, motion models and learned association. IoU, NMS, precision/recall and the assignment step shown here are exactly what those systems compute.

Check yourself

Eduspheria wiki · Applied AI, Classical computer vision

0 / 5 answered

  1. 1Box A is [0,0,4,4] and box B is [2,2,6,6] (x, y, w, h). What is their IoU?
    Numeric answer
  2. 2Raising the NMS IoU threshold from 0.3 to 0.7 will generally…
    Multiple choice
  3. 3A detection counts as a true positive during evaluation when its class matches and its IoU with a ground-truth box exceeds the threshold.
    True / false
  4. 4Which classical algorithm solves the assignment between predicted track positions and new detections?
    Short answer
  5. 5A tracker has 8 true positives, 2 false positives and 3 false negatives. What is its precision?
    Numeric answer

From the exam paper

Modeled on NITJ AI-502, End-Sem May 2025

0 / 5 answered

  1. 1How does YOLO treat object detection differently from two-stage detectors such as Faster R-CNN?
    Multiple choice
  2. 2Why is the Region Proposal Network of Faster R-CNN preferred over selective search?
    Multiple choice
  3. 3Selective search is a trainable neural network that adapts its proposals to the detection task.
    True / false
  4. 4In Faster R-CNN, what does the abbreviation RPN stand for?
    Short answer
  5. 5Which greedy post-processing step removes duplicate overlapping detections by keeping the highest-scoring box and suppressing its neighbours?
    Short answer

Where next: frames arrive one after another, and the next chapter treats the sequence itself as the signal — background modelling and motion.