Wiki
Core13 min read

Activity and behaviour recognition

Recognising actions over time: per-frame scores, temporal windows, smoothing, and the two-stream / 3-D convolutional architectures that made video classification work.

A single frame can tell you someone is standing. Only a sequence tells you they are walking, running, waving or falling. Activity recognition is the problem of assigning a label to a stretch of video, and its central difficulty is temporal: the evidence is spread across time, and a classifier that scores each frame independently will flicker between wrong answers.

The widget makes that flicker visible and shows the standard cure — smoothing the per-frame scores over a temporal window before taking the argmax.

Classification plus temporal structure

Motion gives you features per frame; recognition needs a decision per segment. The gap is filled by a temporal model — a sliding window, a recurrent state, a temporal convolution, or an HMM/CRF decoder. Raw per-frame argmax is almost never the answer you want.

A per-frame classifier flickers between classes; a temporal moving average cleans it up. Widen the window and watch the frame-level accuracy change.

ground truth
idle
walking
running
waving
raw pred
smoothed pred
waving
1.93
walking
0.49
running
0.31
idle
0.10
raw accuracy
86.7%
smoothed accuracy
96.7%
frame class
waving
smoothed class
walking

Synthetic per-frame scores with injected flicker, real moving-average smoothing and accuracy counts. Production systems add temporal convolutions, HMM/CRF decoding or recurrent/temporal-attention heads on top of exactly this signal.

Formulating the problem

There are two classic framings. Segmented recognition takes a clip already trimmed to contain one action and predicts its class. Temporal detection takes a long video and must find when each action occurs, producing intervals with labels. The widget sits in between: a fixed-length timeline of frames each carrying class scores, from which you derive per-frame and smoothed predictions.

Learning spatiotemporal features

  • Two-stream networks. One CNN sees a single RGB frame (appearance), a second sees stacked optical-flow fields (motion). Their scores are fused. The insight is that appearance and motion are complementary and can be learned separately.
  • 3-D convolutions. Replace 2-D spatial kernels with 3-D kernels that also slide over time, so the network sees local motion directly. C3D and its successors are the natural extension of the convolution you already know.
  • I3D and inflating. Take a strong image model and "inflate" its 2-D kernels into 3-D ones, initialising from ImageNet; combine with two-stream input. This is the template for most modern action models before Transformers.

Temporal decoding

Given scores st(c)s_t(c) for frame tt and class cc, a simple and surprisingly strong decoder is a moving average followed by argmax:

c^t=arg⁡max⁡c12w+1∑k=−wwst+k(c).\hat{c}_t = \arg\max_c \frac{1}{2w+1} \sum_{k=-w}^{w} s_{t+k}(c).

More principled decoders treat the smoother output as emission probabilities in an HMM or CRF and run Viterbi to enforce valid transitions — the same dynamic-programming idea as tagging, now over time.

Smoothing trades latency for stability

Widening the temporal window suppresses flicker and raises frame accuracy, but it also delays the response to a real change and blurs short actions into their neighbours. There is no free stability: every deployed system chooses a latency it can tolerate and a minimum action duration it must resolve.

Illustrative vs real

The widget uses synthetic per-frame scores with injected flicker; no video is decoded and no network runs. Real pipelines compute features with 2-D or 3-D CNNs (sometimes with optical flow), train on datasets such as Kinetics or UCF101, and evaluate segment-level accuracy or temporal-IoU detection metrics. The moving-average/Viterbi decoding and the latency–stability trade-off are genuine.

Check yourself

Eduspheria wiki · Applied AI, Video analytics

0 / 5 answered

  1. 1A moving average over 5 frames has class scores 0.2, 0.8, 0.9, 0.7, 0.4. What is the smoothed score?
    Numeric answer
  2. 2In a two-stream network, what does the second stream consume?
    Multiple choice
  3. 33-D convolutions slide their kernels over both space and time.
    True / false
  4. 4What does widening the temporal smoothing window do to the response to a genuine action change?
    Short answer
  5. 5Which task requires finding when actions occur in a long untrimmed video?
    Multiple choice

From the assignment paper

Modeled on NITJ AI-604, Assignment/Quiz

0 / 5 answered

  1. 1What does dense optical flow compute across an image sequence?
    Multiple choice
  2. 2In multimodal human activity recognition, which description matches late fusion?
    Multiple choice
  3. 3The stance phase occupies approximately what percentage of the gait cycle? Enter the percentage.
    %
    Numeric answer
  4. 4Which deep learning model is best suited to tracking motion over time from sequential human pose data?
    Multiple choice
  5. 5Which correlation-filter tracker is efficient because it uses Fourier transforms for fast correlation?
    Short answer

Where next: the same temporal machinery, applied to recognising who — from the geometry of a face and the rhythm of a walk.