Metrics and imbalance
Accuracy is the most confidently-quoting, least informative number in ML. Here's what actually happens when 1% of data matters.
Here is the setup that ends more ML projects than any algorithm: train a classifier where 99% of cases are the negative class ("not fraud, not diseased, not churning"). A model that predicts "never" on everything gets 99% accuracy. It's also worthless — the 99% and the 1% are the same performance number.
What accuracy cannot count
Accuracy counts one error class, so it counts cheap errors and rare errors as equal. When the rare class is the reason you built the model at all, accuracy is not a metric — it's a decoy.
200 predictions scored 0-1; only 2 are truly positive. Slide the alarm threshold.
Red ticks = the 2 true positives (one carefully hidden below 0.5). Watch the trap: at threshold ≈ 0.5 the model shows 100% accuracy while catching nothing. Every reachable recall costs its own precision; the right spot is a business call (cost of missing a fraud vs. cost of reviewing one). There is no free number here — only trade-offs you now get to read honestly.
The confusion matrix, armed
Four counts, each with a name and a real-world cost:
- True positives (TP) — the rare class you caught. The whole point.
- False positives (FP) — alarms your model screamed for nothing. The cost is operational — reviews, chases, sent alerts — and paid every time.
- False negatives (FN) — the fraud/illness/churnd you missed. Often the expensive part; often invisible until much later.
- True negatives — everything else. The number accuracy is mostly made of.
From those, the two views that matter:
- Precision = TP/(TP+FP): "when we raise an alarm, how often were we right?" Spend-preserving — high precision protects your response team from noise.
- Recall = TP/(TP+FN): "of all the real cases, how many did we catch?" Coverage-preserving — high recall protects against blind spots.
They trade against each other — through the classifier's threshold, which is what the artifact's dial is. No classifier escapes the trade: precision goes up as recall comes down, and where to sit is a business decision, not a technical one.
What F1 (and other summaries) hide
You'll meet F1 — the harmonic mean of precision and recall, a convenient compromise number. It's fine for experiments; remember that collapsing two costs into one number erases the asymmetry — in fraud, missing one case (FN) is not "worth" one wasted review (FP). Where the asymmetry is known and large, weight the errors directly or pick a threshold from the cost curve — never from the F1 score alone.
Illustrative vs real
200 rows, a fixed 1%-positive dataset here. Real shops also report PR-AUC (area under the precision-recall curve, robust under imbalance — accuracy-adjacent ROC metrics flatten and lie here), calibration curves, and threshold cost curves per use case.
Where next: regularization — the dial that builds the bias you want on purpose.