Monitoring and drift

A model is a snapshot of a relationship, and the world does not hold still. The population that uses your product changes; a vendor redefines a column; a pandemic rewrites what "normal" means. Once a model is live, the question is no longer "how accurate is it?" but "is it still accurate, and how would I know?" Monitoring is the instrumentation that answers both.

The monitor below compares a reference distribution to an incoming one and computes the population stability index (PSI), the standard drift alarm. Shift the incoming mean or variance and watch PSI cross the usual 0.1 and 0.2 thresholds; the timeline shows the same signal arriving window by window.

Measuring drift #

For a single feature, the population stability index bins both distributions and sums a symmetric divergence:

$$\mathrm{PSI} = \sum_{b} \left(p_b^{\text{current}} - p_b^{\text{reference}}\right) \ln\!\left(\frac{p_b^{\text{current}}}{p_b^{\text{reference}}}\right),$$

with the conventional reading: below $0.1$ stable, $0.1$–$0.2$ moderate shift worth investigating, above $0.2$ act. The KS test and the $\chi^2$ test serve the same purpose for continuous and categorical variables respectively. For the data as a whole, multivariate tests compare the full feature distribution, which matters because individually innocent shifts can combine into a changed input region.

There is a subtle line to hold: a feature can drift without hurting the model (the model may ignore it), and the model can degrade without any input drift at all (that is concept drift, visible only against labels). Input drift is an early, cheap warning; it is not the outcome.

What to monitor #

• Service health — latency, error rate, throughput, and the fraction of requests whose inputs fail validation.
• Data quality — the pipeline tests from the data-engineering chapter, applied to the live feed.
• Prediction distribution — a sudden shift in the score distribution often signals an input problem before labels arrive.
• Feature drift — PSI or KS per input feature.
• Outcome and labels — accuracy, calibration, business metric, and subpopulation slices, once ground truth arrives.

Label delay is the practical trap: churn or default may only be known weeks after the prediction, so outcome metrics lag. Proxy and input signals are what keep you from flying blind in between.

Responding to drift #

Retraining is not the only answer and often not the best one. The response ladder runs: investigate the data pipeline first, then recalibrate, then retrain on recent data, then consider a genuinely different model. A recurring seasonal pattern may need a model that encodes the season, not a fresh fit every cycle. And every response must be validated against the incumbent before it ships — an automatic retraining loop with no evaluation gate will eventually deploy a worse model on drifted data.

Check yourself

Eduspheria wiki · Data, MLOps & Deployment, MLOps

0 / 5 answered

1. 1A feature has reference proportions [0.5, 0.5] and current proportions [0.6, 0.4]. Compute PSI to three decimal places.

   Check Hint Show answerNumeric answer
2. 2The input distribution is unchanged but model accuracy has fallen. What is this?

   A.Concept driftB.Upstream breakageC.Data driftD.Overfitting

   Check Show answerMultiple choice
3. 3Which standard drift statistic bins a feature's reference and current distributions and sums a symmetric divergence?

   Check Show answerShort answer
4. 4A feature showing input drift always means the model's accuracy has degraded.

   TrueFalse

   Check Show answerTrue / false
5. 5Why are outcome metrics harder to monitor than input metrics in production?

   A.They need no thresholdsB.They ignore subpopulationsC.They are always encryptedD.Ground-truth labels arrive with a delay

   Check Show answerMultiple choice

From the exam paper

Modeled on NITJ AI-511, End-Sem December 2024

0 / 5 answered

1. 1In the four-principle view of Model Risk Management (model definition, risk governance, lifecycle management, and one more), what is the fourth principle, which demands independent review and challenge of a model?

   Check Show answerShort answer
2. 2In financial regulation, what does the term model risk refer to?

   A.The risk that training data is stored unencryptedB.The chance that a server hosting the model goes offlineC.The potential for adverse outcomes from decisions based on a model that is wrong, misused, or built on flawed assumptionsD.The risk of writing more code than the team can review

   Check Show answerMultiple choice
3. 3How does a canary deployment reduce risk when a new model version is rolled out?

   A.It removes the previous version immediately to avoid confusionB.It retrains the incumbent model on production data before switchingC.It routes a small slice of live traffic to the new version, monitors it against the incumbent, and expands or rolls back based on the resultD.It sends all production traffic to the new version at once so the switch is fast

   Check Show answerMultiple choice
4. 4Poisoning attacks corrupt a model by manipulating its training data, whereas adversarial attacks craft malicious inputs at inference time to fool an already-trained model.

   TrueFalse

   Check Show answerTrue / false
5. 5What MLOps practice establishes policies, roles, controls and audit trails so that model decisions stay compliant, ethical and accountable?

   Check Show answerShort answer

Where next: time series — the one setting where observation order is the whole problem, and a random train-test split is a mistake.