Wiki
Core8 min read

Metrics and imbalance

Accuracy is the most confidently-quoting, least informative number in ML. Here's what actually happens when 1% of data matters.

Here is the setup that ends more ML projects than any algorithm: train a classifier where 99% of cases are the negative class ("not fraud, not diseased, not churning"). A model that predicts "never" on everything gets 99% accuracy. It's also worthless — the 99% and the 1% are the same performance number.

What accuracy cannot count

Accuracy counts one error class, so it counts cheap errors and rare errors as equal. When the rare class is the reason you built the model at all, accuracy is not a metric — it's a decoy.

200 predictions scored 0-1; only 2 are truly positive. Slide the alarm threshold.

TP 0FP 0FN 2accuracy 99.0%precision —recall 0%

Red ticks = the 2 true positives (both sitting below 0.5). Watch the trap: at threshold ≈ 0.5 the model shows 99% accuracy while catching nothing — with only 2 positives in 200 rows, 99% is the ceiling for zero recall, not 100%, and predicting "always negative" gets you there for free. Every reachable recall costs its own precision; the right spot is a business call (cost of missing a fraud vs. cost of reviewing one). There is no free number here — only trade-offs you now get to read honestly.

The confusion matrix, armed

Four counts, each with a name and a real-world cost:

  • True positives (TP) — the rare class you caught. The whole point.
  • False positives (FP) — alarms your model screamed for nothing. The cost is operational — reviews, chases, sent alerts — and paid every time.
  • False negatives (FN) — the fraud/illness/churnd you missed. Often the expensive part; often invisible until much later.
  • True negatives — everything else. The number accuracy is mostly made of.

From those, the two views that matter:

  • Precision = TP/(TP+FP): "when we raise an alarm, how often were we right?" Spend-preserving — high precision protects your response team from noise.
  • Recall = TP/(TP+FN): "of all the real cases, how many did we catch?" Coverage-preserving — high recall protects against blind spots.

They trade against each other — through the classifier's threshold, which is what the artifact's dial is. No classifier escapes the trade: precision goes up as recall comes down, and where to sit is a business decision, not a technical one.

What F1 (and other summaries) hide

You'll meet F1 — the harmonic mean of precision and recall, a convenient compromise number. It's fine for experiments; remember that collapsing two costs into one number erases the asymmetry — in fraud, missing one case (FN) is not "worth" one wasted review (FP). Where the asymmetry is known and large, weight the errors directly or pick a threshold from the cost curve — never from the F1 score alone.

Illustrative vs real

200 rows, a fixed 1%-positive dataset here. Real shops also report PR-AUC (area under the precision-recall curve, robust under imbalance — accuracy-adjacent ROC metrics flatten and lie here), calibration curves, and threshold cost curves per use case.

From the exam paper

Exam-modeled · End-Sem June 2025

0 / 5 answered

  1. 1A system detects unauthorized access in surveillance footage, and the main concern is missing a real intrusion. Which metric is most appropriate?
    Multiple choice
  2. 2What is the alternative name for recall that is also called the true positive rate?
    Short answer
  3. 3A detector catches 90 of the 100 real intrusions in a test set. What is its recall?
    Numeric answer
  4. 4Because intrusions are rare, a model with very high accuracy is automatically reliable for intrusion detection.
    True / false
  5. 5Which confusion-matrix quantity is increased when the detector misses an actual intrusion?
    Multiple choice

Where next: regularization — the dial that builds the bias you want on purpose.