Wiki
Core11 min read

The ML lifecycle

Framing, data, modelling, deployment, monitoring — a loop with memory, where shipping is the midpoint rather than the finish line.

The moment a model goes live is not the end of a project. From that moment it begins to age: user behaviour shifts, upstream data definitions change, the world it was trained on quietly stops being the world it runs in. A shipped model decays, and the lifecycle exists to notice and respond before anyone else does.

The loop below has memory. Performance slips a little each cycle; finishing Monitoring decides whether the decay is severe enough to retrain. Step around it and watch the sawtooth — the shape most production models actually live on, rather than the clean monotone curve of a course exercise.

Step around the loop — monitoring decides when performance has decayed enough to retrain

FrameDataModelDeployMonitorv1perf 0.90

Performance over cycles

version 1 · 0 retrains · current perf 0.900

The loop is real; the decay numbers are invented. In practice you would trigger retraining from a monitored metric crossing a threshold, and you would weigh the cost of retraining against the benefit — a model that decays gracefully may not need a new version every cycle.

Shipping is the midpoint

A trained model with no monitoring is a dependent you cannot see. The lifecycle is a loop, not a pipeline, because deployment is the point where you start collecting the information that tells you what to fix next. Version the data, the code, and the model together — a model that cannot be reproduced cannot be debugged.

The stages and what each one owes the next

  • Problem framing — define the decision, the metric, and the operating constraints (latency, cost, fairness) before touching data.
  • Data — collection, labelling, splitting, and versioning. This is where most of the time and most of the risk live.
  • Modelling — train a baseline first, then iterate; evaluation is part of modelling, not an afterthought.
  • Deployment — package, serve, and start with a limited blast radius.
  • Monitoring — watch data quality, predictions, and outcomes, not just uptime.
  • Retraining — triggered by decay, drift, or a scheduled cadence, and only if it demonstrably helps.

Technical debt is the hidden stage

The famous diagram from Sculley and colleagues shows a tiny "ML code" box surrounded by configuration, data collection, feature extraction, serving infrastructure, monitoring, and glue. The debt accumulates in the glue: undeclared consumers of a feature, entanglements where changing one thing changes everything, configuration debt, and pipeline jungles. The countermeasure is engineering discipline applied to ML specifically — versioned data, automated tests, declarative configuration, and observability — which is what the next two lessons detail.

Reproducibility and experimentation

Two habits keep the loop sane. First, everything is versioned: code in git, data by snapshot or content hash, model artifacts and the exact environment. Second, experiments are tracked: which data, which parameters, which commit, and what metric, so a result can be re-derived rather than remembered. Without these, the loop turns into folklore, and "why did last quarter's model do better?" becomes unanswerable.

Careful

Continuous retraining is not automatically continuous improvement. A retrain that ships a worse model is a regression, and an automatic pipeline that retrains on drifted or corrupted data will happily deploy the damage. Gate retraining behind evaluation against the incumbent, keep the previous version warm for rollback, and require a monitored metric to justify promotion.

Illustrative vs real

The loop's decay rate and retraining threshold are set by sliders so the behaviour is visible in seconds. Real decay is measured with statistical tests over weeks, and the decision to retrain trades accuracy against compute cost and risk. The stage ordering is the genuine lifecycle; the numbers are a teaching device.

Check yourself

Eduspheria wiki · Data, MLOps & Deployment, MLOps

0 / 4 answered

  1. 1According to the hidden-technical-debt picture, where does most debt accumulate?
    Multiple choice
  2. 2Continuous retraining always improves a deployed model.
    True / false
  3. 3What practice records the data, parameters, commit and metric for each training run so results can be re-derived?
    Short answer
  4. 4Which constraint should be defined during problem framing, before data work begins?
    Multiple choice

From the mid-term paper

Modeled on NITJ AI-511, Mid-Term October 2024

0 / 5 answered

  1. 1What term describes a change in the input distribution P(x) while the relationship from input to target stays the same?
    Short answer
  2. 2What term describes a change in the relationship itself, so that P(y given x) changes after deployment?
    Short answer
  3. 3Which deployment strategy runs a new model alongside the current one and logs its predictions, but does not act on them?
    Multiple choice
  4. 4A canary deployment sends a small fraction of traffic to the new model before a full rollout.
    True / false
  5. 5Which monitored signal most directly suggests that the model's input data has changed?
    Multiple choice

Where next: deployment and serving — getting a versioned model into production without betting the whole user base on it.