Wiki
Core12 min read

Motion and background modelling

Video is a signal over time. Frame differencing and running-average background models separate what moves from what stays still — with real failure modes.

A still image is a snapshot; a video is a function of time. Motion is what changes between consecutive frames, and detecting it is the first step of almost every video analytics pipeline: count people, flag intrusions, measure traffic. The naive idea — subtract one frame from the next — almost works, and understanding exactly why it fails is the whole lesson.

The problem is that "what moves" is not the same as "what differs". A stationary object that turns on a light differs without moving; a person in camouflage moves without differing much. The right target is a background model: an estimate of the scene without the things you care about, against which each frame is compared.

What is the background?

There is no single background — only a model of it that you maintain over time. Run a slow exponential average of the pixels and it converges to the scene's persistent appearance. A pixel far from that model right now is foreground. Everything hard about the problem is in how you update the model so that genuine changes get absorbed and moving things do not.

Scrub through the clip and switch methods; the current frame, the background estimate and the foreground mask are all computed live.

A bright disc moves across a noisy scene. Compare frame differencing, a running average, and a foreground-gated background update.

current frame

background model

foreground mask

foreground pixels

45

method

selective

Real per-pixel arithmetic on a synthetic clip. Frame differencing only flags change and leaves nothing at rest; the plain running average smears the moving object into the background; the selective update is the preprocessing half of Stauffer–Grimson-style mixture-of-Gaussians background modelling.

Three models, three failures

Let ItI_t be the frame at time tt and BtB_t the background estimate.

  • Frame differencing. Foreground where ∣It−It−1∣>τ|I_t - I_{t-1}| > \tau. Simple and fast, but only flags change: a slow-moving object almost vanishes and a stationary one leaves nothing behind.
  • Running average. Update Bt=(1−α)Bt−1+αItB_t = (1-\alpha) B_{t-1} + \alpha I_t, then threshold ∣It−Bt∣>τ|I_t - B_t| > \tau. The background adapts, but a moving object is averaged into it, leaving a smeared ghost and a persistent trail.
  • Selective (gated) update. Update the background only where the pixel is not currently foreground. Moving objects stop polluting the model, at the cost that a real scene change (a parked car, a switched-off lamp) is absorbed only once it stops being foreground.
Bt(x)={(1−α) Bt−1(x)+α It(x)if x is background,Bt−1(x)otherwise.B_t(x) = \begin{cases} (1-\alpha)\,B_{t-1}(x) + \alpha\,I_t(x) & \text{if } x \text{ is background},\\ B_{t-1}(x) & \text{otherwise.} \end{cases}

From running average to mixtures

A single mean per pixel struggles when the scene has multiple stable appearances — leaves that sway, water that ripples, monitors that flicker. Stauffer and Grimson model each pixel with a mixture of Gaussians, assigning each new value to the component it fits best and updating that component's weight, mean and variance. A pixel is background if it matches a high-weight, low-variance component. This handles small persistent motions and adapts its sensitivity per pixel.

What breaks every background model

  • Illumination change. A cloud passing changes every pixel at once.
  • Shadows. The dark region a person casts moves with them and is persistently mislabelled unless colour or gradient cues remove it.
  • Camera motion. If the camera moves, nothing in the image is background in the model's frame; you must first stabilise using the matching techniques of the previous chapter.
  • Intermittent motion. Objects that stop for a while get absorbed into the background and disappear as foreground when they move again.

The model and the foreground are coupled

You need a background model to decide the foreground, but the standard update needs the foreground decision to avoid polluting the model. Get that loop wrong and the background either absorbs the objects you are trying to detect or stops adapting entirely. Every practical system tunes the learning rate α\alpha and threshold τ\tau against this trade-off.

Illustrative vs real

The widget uses a synthetic 24×18 clip with a single moving disc and a simple pixel value, so the arithmetic is transparent. Real systems work on colour or gradients, use multiple Gaussians per pixel, add shadow suppression and morphological cleanup, and often precede all of it with optical flow or ego-motion compensation. The detections above feed the trackers and activity recognisers of the next lesson.

Check yourself

Eduspheria wiki · Applied AI, Video analytics

0 / 5 answered

  1. 1A running average has B = 100 at a pixel; the new frame value is 140 with α = 0.1. What is the updated background?
    Numeric answer
  2. 2Which method flags only *changes* between consecutive frames and leaves a stationary object invisible?
    Multiple choice
  3. 3A selective background update skips updating pixels currently classified as foreground.
    True / false
  4. 4Which per-pixel probabilistic model handles multiple stable appearances such as swaying leaves?
    Short answer
  5. 5A frame's pixel is 100 but the background model says 70. With threshold τ = 20, is the pixel foreground (1) or background (0)? Enter 1 or 0.
    Numeric answer

From the exam paper

Modeled on NITJ AI-604, End-Sem May 2025

0 / 5 answered

  1. 1A video runs at 30 fps for 4 minutes. What is the total number of frames?
    Numeric answer
  2. 2From that 4-minute, 30 fps video, how many frames are extracted if one frame is taken every second?
    Numeric answer
  3. 3If one frame is extracted per second, what is the time interval between consecutive extracted frames, in seconds?
    Numeric answer
  4. 4A clip contains 1800 frames at 25 fps. What is its duration in seconds?
    Multiple choice
  5. 5At 30 fps, a 2-minute clip contains 3600 frames.
    True / false

Where next: once motion is isolated, the next question is what the motion means — recognising activities and behaviours over time.