
Operating a model: monitoring, drift and rollback · Standard gear
As of 18 Nov 2026. Reviewed and published, version 1.0.0, part of How AI Works, the AiGovOps Foundation's planned free course. It is not a credential. The research is dated and true of what those papers and interviews studied. No real operating incident is used as a case: the worked example is a made-up record, labelled as such. Where we do not know something, we say so.
You will be able to:
- start from a system's Level 300 hold card (holder, thresholds, stop) and add versioned monitoring that catches input shift and performance decay even when the true answers (labels) arrive weeks late, and name the threshold that pages the holder;
- write a release and rollback plan that versions the model, its data, its code and its configuration together, stages the rollout and names who can stop it;
- spot three kinds of hidden technical debt in a machine-learning (ML) system (entanglement, hidden feedback loops and undeclared consumers) and name one control for each.
Big Widget Company, its people, its systems and every number in its record are made up for teaching.
The made-up record used in this module
Big Widget's dealer demand forecast predicts each dealer's widget orders for the coming week. The parts-planning team orders parts from it, and the pricing tool reads it too. A dealer's orders for a week are only final six weeks later, so the true answer for any forecast (its label) arrives six weeks after the forecast was used.
| Day | What happened (made up) |
|---|---|
| 0 | Release R13 starts a staged rollout: a new model version (fc-13) and a new feature pipeline (fp-13) that adds a "dealer region" input. Shadow for 7 days, then 10% of dealers for 7 days, then all dealers. |
| 14 | The holder says yes to R13 serving all dealers. No label for any R13 forecast has arrived yet, so that yes rests on early checks, not on measured accuracy. |
| 15 | A new dealer region, Region 9, opens with 40 dealers. The staged rollout never saw it. |
| 16 | The input monitor fails on two daily batches in a row; Region 9 forecasts are 2.6 times those of comparable dealers. The holder is paged. |
| 16 | The holder holds large Region 9 parts orders and orders a rollback. A first attempt swaps only the model weights back to fc-12 and fails. The full rollback to release R12 succeeds. The hold on large Region 9 orders stays. |
| 57 onward | Final orders arrive, six weeks late, for the Region 9 forecasts R13 made before it was withdrawn: fc-13 had over-forecast Region 9 about threefold. |
Each section below uses one part of this record.
1. Monitoring that works before the labels arrive
The misconception this corrects: "A model that passed its tests stays good once deployed."
Tests before release check the model against data from the past. The world it serves keeps moving. Two Google teams said so in the papers this module leans on most. D. Sculley and colleagues (NeurIPS 2015) wrote that unit tests and end-to-end tests are valuable, but "in the face of a changing world such tests are not sufficient to provide evidence that a system is working as intended". Their answer: "Comprehensive live monitoring of system behavior in real time combined with automated response is critical for long-term system reliability."1 Eric Breck and colleagues (IEEE Big Data 2017) put it as: "It is crucial to know not just that your ML system worked correctly at launch, but that it continues to work correctly over time."2
The failure is usually quiet. Stephan Rabanser, Stephan Günnemann and Zachary Lipton (NeurIPS 2019) open their study by noting that ML systems "tend to fail silently" when their inputs change, and that "in practice, ML pipelines rarely inspect incoming data for signs of distribution shift".3
Start from the hold card
At Level 300 you wrote a hold card for a running system: a named holder, the thresholds that matter, and a stop that fails closed. Monitoring at Level 400 does not replace that card. It feeds it: every alert in your monitoring plan must say which threshold on the card it measures and who it pages. Keep the card's provenance label as it came from Level 300 (made at 300, content-free, and not authenticated at source unless it was signed when made).
The US National Institute of Standards and Technology (NIST) AI Risk Management Framework (January 2023) asks for the same link between monitoring and named people. Under its Manage function, "Post-deployment AI system monitoring plans are implemented", including mechanisms for "appeal and override, decommissioning, incident response, recovery, and change management" (MANAGE 4.1).4
The problem of late labels
The obvious monitor is accuracy: compare each forecast with what really happened. That needs labels. Breck and colleagues warn that "it is not always possible to know the correct labels even shortly after serving time".2 Shreya Shankar and colleagues interviewed 18 ML engineers in 2022 (a preprint, not peer-reviewed). Many said that "ground-truth labels for live predictions often arrived after a delay, which could vary unpredictably". One engineer told them feedback "is always delayed by at least 2 weeks".5
At Big Widget, labels arrive after six weeks. An accuracy alarm alone could not have rung before Day 57, six weeks after Region 9 opened. So the plan needs signals that do not wait for labels.
Three layers of signal
| Layer | What it watches | Needs labels? | What the sources say |
|---|---|---|---|
| 1. Inputs | Do today's inputs look like the data the model was trained on? Schema checks (types, ranges, allowed values) and a shift test comparing each input's distribution with the training reference. | No | Breck: encode expectations about the data "in a schema", then "measure whether data matches the schema and alert when they diverge significantly" (Data 1, Monitor 2).2 Rabanser: test whether the source and target distributions differ.3 |
| 2. Outputs | Do the model's predictions, slice by slice, look like a reference: its own launch-time predictions for that slice, or today's predictions for comparable units? Action limits on what the system does. | No | The output comparison is this course's label-free proxy, not a check the sources define. Action limits are Sculley's: if a limit is hit, "automated alerts should fire and trigger manual intervention or investigation."1 |
| 3. Outcomes | When labels arrive, is quality still inside the bounds set at launch, overall and by slice? Prediction bias by slice. | Yes, late | Breck: thresholds for acceptable quality, for example "based on bounds of quality at the launch of the initial system", and "a responder should be notified immediately if quality drifts outside that threshold"; watch "both dramatic and slow-leak regressions" (Monitor 7).2 Sculley's prediction bias compares predictions with observed labels: in a working system "the distribution of predicted labels is equal to the distribution of observed labels", and slicing this "by various dimensions" isolates issues.1 |
Layers 1 and 2 are the early warning. Layer 3 is the proof, and it comes late. Keep the two kinds of output check apart. A label-free output check (Layer 2) can run today, but it only says the forecasts have moved. Prediction bias (Layer 3) says whether they were right, and it has to wait for labels. Two forecasts can look identical today and still turn out very differently once the orders are final. Two cautions:
- Even the label-based check is a smoke alarm, not a verdict. Sculley and colleagues say matching distributions "is by no means a comprehensive test", since a model that predicts the average every time would pass it.1 A label-free output check is weaker still. A steady output distribution does not show that the forecasts are good.
- Shift is not always harm. Rabanser and colleagues note that "in practice distributions shift constantly, and often these changes are benign"; without labels you cannot compute the error on the new data directly.3 An input alert says "look now", not "the model is wrong".
How well can a shift test see? On their image datasets (MNIST and CIFAR-10, with simulated shifts), Rabanser and colleagues found large shifts could on average be detected better than chance with only 20 samples using their best method, while small and medium shifts needed "orders of magnitude more samples". Shifts affecting only 10% of the data were hard to detect.3 Those are results for images, not for a demand forecast. The lesson that carries over is the shape: big shifts show up fast; small or partial ones need time or more data, which is why Layer 3 still matters.
Breck and colleagues add two monitors that catch common causes of shift at the source: dependency changes ("Partial outages, version upgrades, and other changes in the source system can radically change the feature's meaning") and training/serving skew, where the code that computes a feature in production differs from the code that computed it in training. They call skew "Perhaps the most important and least implemented test".2
The threshold that pages the holder
Every alert either pages a person, opens a ticket, or is deleted. If too many alerts page, people stop reading them. In Shankar's interviews, false-positive alerts were "The most commonly discussed pain point", and a surplus of them "led to fatigue and silencing of alerts, which could miss actual performance drops".5 Breck and colleagues say alert thresholds need "careful tuning" to balance false positives against false negatives.2
So this course's rule is: name, on the hold card, one paging threshold per layer, and send everything else to a ticket. A threshold is a choice, and a person owns it. Before you trust one, replay it over calm past periods and count how often it would have paged. If you have not done that, the honest answer to "how often will this page falsely?" is "not known yet", with the name of whoever will find out. Sculley and colleagues also warn that "Relying on human intervention in response to alert pages is one strategy, but can be brittle for time-sensitive issues."1 That is why the Level 300 stop exists: the page tells the holder; the stop acts without waiting for them. A model's forecast never places a large order by itself. Anything it cannot undo waits for a person.
Keep the jobs apart. The monitor detects and informs. The stop holds. A named person decides whether to release a hold, advance a release or roll back. Neither the model nor the monitor ever says yes, and there is no direct path from a forecast to a consequential action such as a large parts order.
Version the monitor
A monitor compares live data with a reference: the training data, its schema and the launch-time quality bounds. Those belong to one model version. When the model changes, the reference must change with it, and when the model rolls back, the monitor must roll back too. Sculley and colleagues make the general point for configuration: "Configurations should undergo a full code review and be checked into a repository", and "It should be easy to see, visually, the difference in configuration between two models."1 In this course, the monitor's settings (reference data version, features watched, thresholds, who is paged) are part of the release's configuration and carry its version number. That is the course's practice, built on the sources; it is not a rule from them.
At Big Widget Company (made up)
The forecast's hold card names Dana Reyes, demand-planning lead, as holder. R13's monitor settings (mon-13) say:
- Layer 1 page: the daily shift test fails on two daily batches in a row. One failed batch opens a ticket.
- Layer 2 page: any region's mean forecast is more than twice that of comparable dealers in other regions. This is an action limit Dana chose, not the result of a statistical test.
- Layer 3 page: when labels arrive, any region's forecast error is outside the launch bounds.
- Who is paged: Dana Reyes; the on-call forecast engineer gets a copy.
- The stop (from the 300 card): parts orders driven by a held forecast above the planner's cap wait for a person.
These settings are made up for the story. They are not recommended values.
On Day 15, Region 9 opens. Its dealers are a category the model never saw, and they have almost no order history. On Day 16 the shift test fails for the second batch in a row (Layer 1), and Region 9's forecasts are 2.6 times those of comparable dealers (Layer 2). Dana is paged on Day 16, not Day 57. The early signal was the inputs and outputs, not accuracy, because accuracy could not be known for six weeks.
What was known on Day 16: the inputs were new and the forecasts were unusual. What was not known yet: whether the forecasts were wrong. That question had an owner (Dana) and a date (final orders from Day 57). The page was a reason to look and to hold large orders, not proof of error.
2. A release and rollback plan
Figure data as text
Four parts, one release ID
| Part | What it shows |
|---|---|
| MODEL [s2] | Exact weights and spec. Know the exact code that was run to produce a given model. |
| DATA [s1] | Training snapshot and signals. Keep versioned copies of input signals another team owns. |
| CODE [s2] | Feature pipeline and serving. A newer model can need code an older server lacks. |
| CONFIGURATION [s1] | Thresholds, features, monitors. Reviewed and checked into a repository. |
| ONE ID (COURSE) | One release ID. A rollback restores a release ID, never one part. |
The misconception this corrects: "Rolling back a model is the same as rolling back code."
Breck and colleagues frame it this way: "training data needs testing like code, and a trained ML model needs production practices like a binary does, such as debuggability, rollbacks and monitoring".2 But a model is not only its weights. Its behaviour depends on the data it was trained on, the code that computes its inputs, and the settings that turn its scores into actions. Sculley and colleagues call this "the CACE principle: Changing Anything Changes Everything", and they apply it not only to inputs but to "hyper-parameters, learning settings, sampling methods, convergence thresholds, data selection, and essentially every other possible tweak".1 Roll back one of those and not the others, and you are running a combination nobody tested.
Version four things together
| Part | What to pin | Why (source) |
|---|---|---|
| Model | The exact weights, and the model specification that produced them | Breck: "it's crucial to know the exact code that was run to produce a given learned model" (Model 1).2 |
| Data | The training-data snapshot, and frozen copies of any input signals another team owns | Sculley: for unstable data dependencies, "create a versioned copy of a given signal", noting that versioning "carries its own costs", such as staleness.1 |
| Code | The feature pipeline and the serving code | Breck: a newer model can need code an older server lacks, so the server "will refuse to load the model" (Infra 6); training and serving code can drift apart (Monitor 3).2 |
| Configuration | Thresholds, the feature list, the monitor settings and the paging rules | Sculley: configurations reviewed and "checked into a repository"; a model's fixed threshold "may be invalid" once the model updates on new data.1 |
This course's rule, built on those reasons: give the four one release ID. A rollback restores a release ID, never one part. The papers give the reasons; the single ID is the course's way of acting on them. Shankar's interviewees described deployment as including "keeping records of any change to production in case of a necessary rollback", and they responded to buggy models by "switching the model to a simpler, historical, or retrained version".5
Stage the rollout
Do not switch everyone at once. Breck and colleagues recommend turning up new models "gradually, running old and new models concurrently, with new models only seeing a small fraction of traffic, gradually increased as the new model is observed to behave sanely" (Infra 6). Before that, an automated check "must either bless the model or veto it", comparing it with "the previous version of the model while setting tighter thresholds" to catch sudden drops (Infra 4).2
Shankar's interviewees used stages under many names ("test, dev, canary, staging, shadow, A/B"), usually "between one and four" of them. In a shadow stage, predictions are made live but not shown to users. One limit matters for this module: where the product has a feedback loop, the authors note it is "impossible to evaluate the model in shadow mode because users do not interact with shadow predictions".5
Be exact about what each stage can show before labels arrive. In this course's reading:
- Shadow can show that the new release loads, serves, passes its input checks and produces plausible outputs. It cannot show what happens when people act on its outputs.
- A small live share limits how many dealers are exposed if the release is bad. Until labels arrive, it shows early signals for that share, not accuracy.
- Neither measures outcome quality while labels are still pending. With six-week labels, every stage at Big Widget ended before a single label came back.
So moving to the next stage before labels arrive is a decision by a named person. It is one of four answers: allow (go ahead), constrain (only this far, for example 10% of dealers), hold (wait for a person or for labels) or deny (no). Record the answer, who gave it and what evidence it rested on. "No page yet" is not evidence that the forecasts are right. A rollout also says nothing about conditions it never met: Region 9 opened after R13 reached every dealer.
Name who can stop it, and practise
The plan names, by person, who can stop the rollout at each stage and who can order a rollback. NIST's framework asks that "Mechanisms are in place and applied, and responsibilities are assigned and understood, to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use" (MANAGE 2.4).4 For Big Widget, that is the holder on the 300 card, plus the on-call engineer, who may roll back without waiting but must tell the holder.
Then practise. Breck and colleagues: "Being able to quickly revert to a previous known-good state is as crucial with ML models as with any other aspect of a serving system. Because rolling back is an emergency procedure, operators should practice doing it normally, when not in emergency conditions" (Infra 7).2 A dry rollback in a sandbox, restoring the previous release ID, is how this course has you prove the plan works.
A rollback restores the system. It does not undo what happened while the bad version ran. Note in the plan which forecasts, orders and later training labels the bad version touched, so the next training run can find them. (That is the course's practice, following from the feedback loops in section 3.)
A rollback also does not make the old release right for a world that has changed. Restoring a training snapshot pins what the model learned from; the model still reads today's live inputs. If something new appeared (a new region, a new sensor, a new kind of customer), the old release has not seen it either.
At Big Widget Company (made up)
R13's plan pinned: model fc-13; data snapshot ds-2026-09; feature pipeline fp-13 and serving build sv-13; configuration cfg-13 with monitor settings mon-13. R12 was the previous known-good release.
On Day 16, before Dana was reached, the on-call engineer swapped the model weights back to fc-12 and left everything else alone. Forecasts stopped: fp-13 was still sending the new "dealer region" input, and fc-12 expected R12's feature list. The serving check rejected the requests, so the 300 stop held large parts orders for a person. That failure was loud, which was lucky; a mismatch that did not change the shape of the inputs could have run silently with skewed features.
The engineer then restored release R12 as a whole: fc-12, fp-12, sv-12, cfg-12 and mon-12, so the monitor compared live data against R12's reference again. Forecasts resumed. The incident record listed the Region 9 forecasts R13 had made, so they could be marked before any retraining. The missing piece in the first attempt is the lesson of this section: a rollback plan that pins the model weights but not the feature pipeline is not a rollback plan.
R12 had never seen Region 9 either. So Dana kept the hold on large Region 9 parts orders after the rollback: R12 was allowed to forecast, but big Region 9 orders still waited for a planner. The rollback restored a known-good release. It did not, by itself, show that R12 was good for Region 9. The hold has an owner and a review date, so it does not quietly become permanent: Dana reviews it when Region 9 labels start to land, from Day 57.
3. Hidden technical debt: the model is the small box
The misconception this corrects: "The model is most of the system."
Sculley and colleagues observed that "developing and deploying ML systems is relatively fast and cheap, but maintaining them over time is difficult and expensive", and that "Hidden debt is dangerous because it compounds silently."1 Their Figure 1 shows the ML code as a small box inside a large system of configuration, data collection, feature extraction, serving and monitoring: "Only a small fraction of real-world ML systems is composed of the ML code". They suggest "a mature system might end up being (at most) 5% machine learning code and (at least) 95% glue code".1 Treat that as their estimate from experience, not a measured figure for every system. The paper names many risk factors; this module teaches three, each with one control.
Entanglement
"Machine learning systems mix signals together, entangling them and making isolation of improvements impossible." Change the distribution of one input, or add or remove one, and "the importance, weights, or use of the remaining" inputs "may all change". "No inputs are ever really independent."1
Control: before release, compare the new version's predictions with the old version's, slice by slice. Sculley and colleagues suggest "detecting changes in prediction behavior as they occur" and say "Metrics that operate on a slice-by-slice basis may also be extremely useful."1 Breck and colleagues make it a release test: thresholds per slice, including an incremental one "compared to the previously released model" (Model 6).2
At Big Widget: fp-13 added only the "dealer region" input, but fc-13 also leaned less on dealer order history, which is why brand-new Region 9 dealers with no history got forecasts far above comparable dealers. A slice-by-slice comparison of fc-13 against fc-12 for low-history dealers would have shown the change before Day 0.
Hidden feedback loops
A live model can end up "influencing [its] own behavior if [it updates] over time". A direct loop is a model that influences "the selection of its own future training data". A hidden loop is harder: "two systems influence each other indirectly through the world". Both are "more difficult to detect and address if they occur gradually over time".1
Control: keep some data out of the loop. For direct loops, Sculley and colleagues suggest "some amount of randomization" or "isolating certain parts of data from being influenced by a given model".1 Shankar's interviewees add the warning above: shadow mode cannot evaluate a model whose users would have reacted to it.5
At Big Widget: the forecast feeds the pricing tool. Pricing offers smaller discounts where it expects strong demand. Dealers facing smaller discounts order less. Those orders become next month's training labels, so the forecast is partly learning from the effect of its own forecasts, through another team's tool. The control, applying the paper's direct-loop idea to this hidden loop (the course's step, not the paper's): a randomly chosen group of dealers whose prices are not set from the forecast. Split that group in two. One part supplies training data the forecast did not cause; the other is kept only for checking the model, never for training it. Being outside the loop does not, by itself, make data a fair test set.
Undeclared consumers
A model's prediction is often "made widely accessible, either at runtime or by writing to files or logs that may later be consumed by other systems". Without access controls, some consumers are undeclared, "silently using the output of a given model as an input to another system". This creates "a hidden tight coupling", and changes to the model will "very likely impact these other parts". Undeclared consumers may also create hidden feedback loops.1
Control: close the door and keep a list. The paper says undeclared consumers are hard to detect "unless the system is specifically designed to guard against this case, for example with access restrictions or strict service-level agreements (SLAs)".1 Breck and colleagues give the reverse duty to consumers: "make sure that the dependent team knows your team is using the data" (Monitor 1).2
At Big Widget: the finance team's warranty-reserve spreadsheet reads the forecast log file. Nobody told the forecast team. When R13 changed the forecasts, finance's reserve moved and nobody knew why. The control: forecast logs readable only by declared consumers, each named in the release plan and told before a release.
A check on the whole system
Breck and colleagues turn these ideas into "28 specific tests and monitoring needs" in four areas (data, model, infrastructure and monitoring). Each test scores half a point if run manually with documented results and a full point if automated; the final score is the minimum of the four area scores, so a system cannot hide a weak area behind a strong one.2 When they interviewed 36 Google teams, several assumed that an upstream data team or a larger serving system would catch their problems; the authors say that can be some protection, but a smaller system's errors may be masked in the noise of the larger one.2
4. Putting it together
A plan for operating one model fits on two pages:
- The hold card, carried from Level 300, with its provenance label: holder, thresholds, stop.
- The monitoring plan: the three layers, the one paging threshold per layer, who is paged, and the monitor settings versioned with the release.12
- The release plan: one release ID pinning model, data, code and configuration; the stages; who can stop each stage; the rollback to the previous release ID, rehearsed.24
- The debt register: for each entanglement, feedback loop and consumer you know of, its control.1
NIST's framework says it is "incumbent on Framework users to continue to apply the MANAGE function to deployed AI systems as methods, contexts, risks, and needs or expectations from relevant AI actors evolve over time".4 The plan is a living record, not a launch document.
The three yeses for a running model
This is the operating part of the AiGovOps Foundation's three yeses, taught in module 100.9: Get to Yes · Stay at Yes · Recover to Yes. Mind the small words: you get to yes, stay at yes and recover to yes. A plan is evidence, not permission. A named person says yes, never the model or the monitor, and every gate answer is one of four: allow (go ahead), constrain (only this far), hold (wait for a person) or deny (no).
- Get to Yes (before each stage of a release). The release gate blesses or vetoes the release ID. Dana, as holder, gives the answer for each stage and records what it rested on. Before labels exist, that answer rests on early checks, and the record says so.
- Stay at Yes (while it runs). The three layers keep checking, and the monitor's reference moves with the release. The first yes is not the last check. Layer 3 still runs when the labels land, even if nothing paged.
- Recover to Yes (when a page fires). Hold what cannot be undone (large parts orders), roll back the whole release ID, list what the bad release touched, and fix the cause. Stopping the bad release is not, by itself, a return to yes. Call it recovered only when the labels or another agreed check show the restored release is inside its bounds for the slices that failed, and Dana agrees. Until then the honest status is "not known yet", with Dana as owner.
What to keep, and what to share. Keep the incident record, the list of affected forecasts and the dealer-level evidence in the team's own records, access-controlled. A receipt of a gate decision holds no private content: the decision, who made it, when, the release ID and references (such as hashes) to the evidence. It never carries forecasts for named dealers, order data, customer details or the contents of the incident notes. In this module the receipt can be an unsigned decision record. A hash in it lets someone confirm the referenced record has not changed; it does not show that the record is right, and it is not permission to publish what the record contains. Modules 400.8 and 400.9 show how to make such a receipt checkable by a stranger.
Before you say yes
The plan is ready for a named person to decide on when you can say each of these, or say plainly that you cannot yet and who will close the gap:
- I know which threshold pages the holder on each layer, and which threshold on the 300 card it measures.
- I know the combined rule's false-page rate from a replay, or I record it as "not known yet" with an owner.
- I can restore the previous release ID as a whole, and we have rehearsed it outside an emergency.
- I know every consumer of the output, and each is named in the release plan.
- I know who answers allow, constrain, hold or deny at each stage, and what evidence each answer rests on.
What this module does not claim
- Shankar and colleagues report velocity, validation and versioning as what governs success in the deployments their 18 interviewees described. That is an interview finding, not a measured cause.5
- Rabanser and colleagues tested shift detection on image data with simulated shifts; their detection numbers do not transfer directly to forecasts or text.3
- The two Google papers describe Google's experience in 2015 and 2017.12
- The one-release-ID rule, the label-free output proxy, the one-page-per-layer rule and the split hold-out group are this course's practice, built on the sources. The papers did not test them.
- The Big Widget thresholds (two failing batches, twice comparable dealers) are made up. This module gives no tested false-page rate for them; that has to be measured on the real system.
- A complete operating plan is not an assurance that a system is safe. It records what is watched, who decides and how to return to a known-good release. Finishing this module describes your practice, not any system's assurance.
Further learning (not sources)
Free course materials for going deeper, opened on 18 Nov 2026 (levels/400/universities.md). They are not citations, and nothing above rests on them: Stanford CS 329S Machine Learning Systems Design (https://stanford-cs329s.github.io/); University of Washington CSE 599W Systems for ML (https://dlsys.cs.washington.edu/); University of Michigan EECS 598 Systems for Generative AI, Winter 2024, public slides and readings, not verified as complete (https://github.com/mosharaf/eecs598/tree/w24-genai).