
The message arrives on a Monday: “the model got worse.” No error, no outage, just a business metric that sags and a stakeholder who wants it fixed by lunch. The reflex is to retrain. That reflex is wrong about as often as it is right, and when it is wrong it burns a day and hides the real fault. This piece is a field diagnosis for that exact situation, organised as symptoms, likely causes and the fix that actually matches each cause. It is built on the diagnosis guidance in the Ops Atlas page about MLOps, with my own notes on how to run it without drama.
Step zero: confirm the problem is real
Before opening any dashboard, check whether the comparison itself is fair. One of the anti-patterns the Atlas lists is comparing unlike windows, for instance a holiday week against an ordinary one, or a month with a promotion against one without. Pull the same weekday and season from last year if you have it. Also check how labels arrive. If true outcomes lag by weeks, as with a cancellation that only becomes visible after a month, then a drop in a quality number may simply reflect a half-matured cohort. Wait, or compare cohorts of equal age.
Symptom one: predictions look different, but the inputs look fine
Likely cause: the model changed, or the way it is called changed. Check the registry first. Which version is live, and did an alias move recently? Check traffic splits and any canary configuration, since a forgotten experiment can route a slice of requests to an old or unproven version.
Fix: if a recent release is the culprit, go back by moving the alias or traffic share to the previous good version. The Atlas is firm that rollback is not done by retraining. Then investigate the release at leisure.
Symptom two: one field suddenly looks odd
Likely cause: an upstream defect. A column arrives empty, a unit changes, an identifier format shifts. The tell is that the change is abrupt and confined to a field or two, and that it coincides with a known upstream event.
Fix: repair it at the source and, if needed, backfill. Do not teach the model to tolerate it. Add a schema check so the next occurrence is caught at the door; the reference implementation in the Atlas pages the on-call person once schema violations persist for fifteen minutes, which is a useful model of a short, specific rule.
Symptom three: offline tests are fine, live results are worse
Likely cause: training-serving skew. The training pipeline and the live pipeline prepare inputs differently. Typical culprits are different text normalisation, different handling of missing values, or a feature computed from a table that refreshes on a different clock.
Diagnosis: replay logged live inputs through the training-side code and compare the outputs. This parity replay is an Atlas staple, and the acceptable skew rate is zero. If you did not log what the model saw at serving time, you now know what to fix first, and the Rules of ML make the same point.
Fix: give each feature one definition, used on both paths. Retraining on top of skew would only fit the model to the bug.
Symptom four: the whole input population has shifted
Likely cause: genuine data drift. The world, the customers or the season changed. The Atlas separates this from a shift in the base rate of outcomes and from a change in the relationship between inputs and outcomes, because each calls for a different response.
- Inputs shifted, labels still available: retrain on fresh data.
- Base rate of outcomes shifted: recalibrate the decision thresholds, which is cheaper than retraining.
- The input-to-outcome relationship changed: you need new labels, new features or a redesign. Retraining on the old design will not help.
Choosing a drift measure without fooling yourself
Drift numbers deserve scepticism, and the Atlas gives concrete reasons. The much-quoted cut-offs for the PSI score, 0.1 and 0.25, rest on convention with no academic backing, and the value itself moves with your binning choices and sample sizes. In one lab the Atlas ran, a numeric field scored 0.108 on PSI and 0.139 on Jensen-Shannon distance, so both flagged it, yet when a reference bin was empty, nudging a tiny smoothing constant swung PSI anywhere between 0.146 and 0.425, while Jensen-Shannon stayed between about 0.132 and 0.142. That stability is why some tools default to it for categorical data above a thousand rows. Statistical tests such as Kolmogorov-Smirnov and chi-square suit small samples; on large samples they flag everything, because tiny differences become statistically significant.
Practical rule: choose one distance, calibrate its alert level on your own history, and watch your handful of critical columns individually. Evidently’s default is a good cautionary example: it calls the dataset drifted only after over half of the columns move, which can bury one broken field that matters a great deal.
Symptom five: quality fell but no input drift shows
Likely cause: concept drift, where the relationship between inputs and outcomes moved. Label-free estimates struggle here. The Atlas quotes NannyML’s own documentation that its performance estimation method will not work under concept drift, so do not lean on an estimate that cannot see this kind of change.
Fix: collect fresh labels, review features and consider whether the problem needs redesign. This is the slowest path, so confirm it by elimination, after the first four symptoms are ruled out.
Symptom six: monitoring is calm but you do not trust it
Likely cause: the monitor itself is stale. The Atlas flags a subtle trap in push-based metrics, where a gateway keeps the last value pushed by a job that has since stopped, so a dead job looks like a healthy flat line. Watch the push timestamp, and alert on statistics jobs that have not run for longer than a day; the reference implementation uses a threshold of twenty-six hours.
Fix: monitor the monitors, and give every check an owner.
A worked scenario
Consider a fictional subscription service whose cancellation-risk model starts flagging far fewer customers on a Tuesday. Walk the flow. The comparison is fair: same weekday, no holiday. No release happened. No single field looks broken. The replay of logged inputs through the training code disagrees on one field, because the live path now receives plan names in capital letters after a front-end change, while the training path expects lower case. That is skew, caused by an upstream formatting change, and the fix is a shared normalisation function on both paths plus a test in the release gate. No retraining was needed, and the whole diagnosis took under an hour because the team had logged the served inputs and kept the replay script. Had they skipped the flow and retrained, the new model would have learned the odd capitals as signal and the problem would have returned in a different shape the following month.
The scenario is invented, but every step in it comes from the practices described above. Its lesson is that the diagnosis flow is cheap and the guess is expensive.
The one-page decision flow
- Is the comparison fair, with equal windows and matured labels?
- Did a release, alias or traffic split change? If yes, roll back first.
- Is one field abnormal? Fix the source.
- Does a replay of served inputs match the training path? If not, fix the skew.
- Has the population shifted? Decide between retraining, recalibrating and redesign by drift type.
- Is monitoring itself alive?
Put that flow in the runbook, and teach new on-call engineers to walk it top to bottom. If you are growing into the role that runs such runbooks, the career notes on moving from a junior to a senior DevOps engineer show how responsibility widens, and the route from sysadmin and support into DevOps covers the on-call habits that carry straight over.
What to record afterwards
Write down the symptom, the cause, the fix and the evidence. Over a quarter, the tally tells you where to invest: if most incidents are skew, build the replay test into your release gate; if most are upstream defects, ask for stronger agreements with the data owners. The Atlas notes that a quarterly rollback rehearsal keeps the exit route usable. Treat incident reviews as the other half of that habit. Retrain when the diagnosis says retrain, and not a minute before.