
Operating a model: monitoring, drift and rollback · Plain gear
As of 18 Nov 2026. Reviewed and published, version 1.0.0, part of How AI Works, the AiGovOps Foundation's planned free course. It is not a credential. The research comes from dated papers, and it is true of what those papers studied. There is no real incident here. Big Widget's story is made up, and says so. If we do not know something, we say so.
You will be able to:
- add warning lights to an AI system's hold card, so you hear about trouble before the real answers come back, and say which warning wakes the holder;
- write a plan for switching to a new AI version, and for switching back, that keeps all the parts together and names who can say stop;
- spot three hidden problems in a system built around an AI, and name one fix for each.
Big Widget Company is made up for this course. So is everyone and every number in its story.
The story: the forecast and the new region
Big Widget has an AI that guesses how many widgets each dealer will order next week. This guess is called the forecast. The parts team buys parts based on it. The pricing tool reads it too.
Here is the catch. A dealer's orders are only final six weeks later. So Big Widget learns whether a forecast was right six weeks after it was used. The right answer is called the label. Here, labels come six weeks late.
One autumn, Big Widget switched to a new version of the forecast. Two weeks later, a new sales region opened: Region 9, with 40 new dealers. The rest of this module follows what happened.
1. Warning lights that do not wait for the answers
The mistake this fixes: "If the AI passed its tests, it stays good."
The tests were run on old data. The world keeps changing. Engineers at Google wrote in 2015 that tests alone cannot show a system is still working "in the face of a changing world". They said live watching, with an automatic response, is key to keeping a system reliable.1 A 2017 paper from Google said it plainly: you need to know the system "continues to work correctly over time".2
The trouble is often silent. A 2019 study said AI systems "tend to fail silently" when their inputs change. It also said that, in practice, AI systems rarely check incoming data for this kind of change.3
Think of a car dashboard
A car does not wait until the engine breaks to warn you. It has lights for low oil, hot engine and low tyre pressure. Each light is an early sign.
An AI system needs early signs too, because the real answers (labels) come late. In a 2022 study, researchers talked with 18 engineers who run AI systems. Many said labels often came late, by an amount that was hard to predict. One said feedback was "always delayed by at least 2 weeks".4 This study was not checked by other experts before it was posted.
Three kinds of warning light
- Are the inputs strange? Check that today's data looks like the data the AI learned from. Are there new kinds of dealer? Missing values? Numbers out of range?23
- Are the guesses strange? Compare the AI's guesses with a "normal" you pick: its own guesses when it was launched, or today's guesses for similar dealers. Do it for each group of dealers. This check is this course's own idea of an early warning. It does not need the real answers, so it cannot tell you the guesses are right. Also set limits on what the AI is allowed to do. If it hits a limit, someone looks.1
- Were the guesses right? When the labels finally arrive, compare the guesses with the real answers, group by group.1 Check that the quality is still as good as it was at launch.2
Lights 1 and 2 come early. Light 3 is the proof, but it comes late. Light 2 only tells you the guesses have moved. Two sets of guesses can look the same today and still turn out very differently.
Two warnings about warning lights. First, a change in the data is not always harm. The 2019 study notes that data changes all the time, and often the change does no damage.3 A light means "look now". It does not mean "the AI is wrong". Second, too many lights is as bad as none. In the 2022 interviews, false alarms were the most talked-about problem. People got so many that they started to ignore them, and so could miss real drops.4
Start from your hold card
At Level 300 you made a hold card. It names one person, the holder, who is responsible for the system. It lists the limits that matter. And it has a stop that makes the system wait safely if something goes wrong. Keep the card's label saying it was made at Level 300.
Now add the warning lights to that card. This course's rule: for each kind of light, pick one warning that wakes the holder (a "page"). Everything else goes on a to-do list. A warning level is a choice, and a person owns it. Before you trust it, try it on past days when nothing went wrong, and count how often it would have woken the holder for nothing. If nobody has done that yet, the honest answer is "not known yet". Say so, and name the person who will find out.
The Google engineers warned that waiting for a person to answer a page can be too slow for urgent problems.1 That is why the stop acts on its own. The AI never places a big order by itself. Anything that cannot be undone waits for a person.
A warning light never decides anything. It tells the holder. The holder decides.
A US standards body, NIST, says the same kind of thing in its 2023 AI risk framework: once a system is running, there should be plans for watching it, for overriding it, and for responding when things go wrong.5
Keep the lights with the version
Each warning light compares today with "normal". But "normal" belongs to one version of the AI. If you change the version, change the lights' idea of normal too. If you switch back, switch the lights back. Google's engineers said settings like these should be reviewed and saved like code.1
What happened at Big Widget (made up)
The hold card names Dana Reyes, the demand-planning lead, as holder. The warning that pages Dana:
- the input check fails two days in a row; or
- any region's forecasts are more than twice those of similar dealers.
Dana picked these levels. They are made up for the story, not advice.
Region 9 opened on Day 15. Its dealers were a new kind the AI had never seen. They had almost no order history. On Day 16 the input check failed for the second day in a row. Region 9's forecasts were 2.6 times those of similar dealers. Dana was paged on Day 16. The labels would not have shown the problem until Day 57 or later.
On Day 16, Dana knew the inputs were new and the guesses were odd. Dana did not know yet if the guesses were wrong. That would be known from Day 57, and finding out was Dana's job. The page was a reason to look, and to hold big orders. It was not proof.
2. A plan for switching, and for switching back
Figure data as text
Four parts, one release ID
| Part | What it shows |
|---|---|
| MODEL [s2] | Exact weights and spec. Know the exact code that was run to produce a given model. |
| DATA [s1] | Training snapshot and signals. Keep versioned copies of input signals another team owns. |
| CODE [s2] | Feature pipeline and serving. A newer model can need code an older server lacks. |
| CONFIGURATION [s1] | Thresholds, features, monitors. Reviewed and checked into a repository. |
| ONE ID (COURSE) | One release ID. A rollback restores a release ID, never one part. |
The mistake this fixes: "Switching back an AI is like undoing a change to a program."
Think of a recipe
A cake depends on the recipe, the ingredients, the oven and the oven setting. Say a new cake comes out badly. You go back to the old recipe, but keep the new flour and the new oven setting. You still get a bad cake, or a different bad cake.
An AI is the same. Its behaviour depends on four things:
- The model: the trained AI itself.
- The data it learned from.
- The code that prepares its inputs and serves its answers.
- The settings: its limits and its warning lights.
The 2015 Google paper calls this "Changing Anything Changes Everything".1 The 2017 paper says being able to go back quickly to "a previous known-good state" is crucial.2 This course turns those two ideas into one rule: keep the four parts together under one version number. When you switch back, switch back all four. The rule is ours, built on the papers; the papers do not say it in these words.
The recipe is only a picture. Going back to the old recipe helps only if the old recipe suits today's kitchen. If something new has turned up, the old version has not seen it either.
Switch a little at a time
Do not give the new version to everyone on day one. Google's 2017 paper says to turn it on for a small share first, run the old and new side by side, and grow the share only while the new one behaves.2 Several companies in the 2022 interviews did this in stages, usually one to four. One stage is "shadow": the new version makes guesses, but nobody sees them.4
Be clear about what each step can show before the real answers come back. A shadow step shows the new version runs and its guesses look sensible. It cannot show what happens when people act on the guesses. A small share limits the damage if the new version is bad. Neither one shows the guesses are right. At Big Widget every step was over long before the first real answer came back.
So moving to the next step is a choice by a named person, never by the AI. There are four answers: go ahead (allow), only this far (constrain; say, 10% of dealers), wait for a person (hold; for example, until the real answers come back) or no (deny). Write down the answer, who gave it and what it was based on. "No warning yet" does not mean "the guesses are good".
Name who can say stop, and practise
The plan names the people who can stop the switch, and who can order a switch back. NIST says roles for switching off or replacing an AI system that is not doing its job should be given out and understood.5
Then practise switching back on a calm day. The 2017 paper says switching back is an emergency step, so people should practise it "when not in emergency conditions".2
Switching back fixes the system. It does not undo the bad guesses it already made. Write down which guesses the bad version made, so they are not used to teach the next version by mistake.
What happened at Big Widget (made up)
The new version, called R13, had a new model and new input code. The input code added a "region" field. On Day 14, Dana said yes to R13 for all dealers. No real answers had come back yet, so that yes rested on the early checks. Region 9 opened the next day, so the steps never tested it.
On Day 16, the engineer on duty switched back only the model to the old one, from R12. It did not work. The new input code was still sending the "region" field. The old model did not expect it. The forecasts stopped, and the stop on the hold card held big parts orders for a person to check.
Then the engineer switched back all four parts to R12: the old model, the old data record, the old code and the old settings, warning lights included. The forecasts came back. The team wrote down every Region 9 forecast R13 had made.
The old version had never seen Region 9 either. So Dana kept the hold: big Region 9 parts orders still waited for a planner to check them. A hold is not forever. Dana wrote down when to look at it again: when Region 9's real answers start to come back, from Day 57.
The lesson: a switch-back plan that saves the old model but not the old input code is not a switch-back plan.
3. Hidden problems: the AI is the small part
The mistake this fixes: "The AI model is most of the system."
The 2015 Google paper drew a picture. The AI code was a small box in the middle. Around it were big boxes for collecting data, preparing it, serving answers, settings and watching. They wrote that only "a small fraction" of real systems is the AI code.1 They also said building an AI system is fairly quick and cheap, but keeping it running is hard and costly, and hidden problems grow quietly.1
Here are three hidden problems, each with one fix.
Tangled inputs
The AI mixes its inputs together. Change one input, and the AI may change how much it uses all the others. The paper says no inputs "are ever really independent".1
Fix: before switching, compare the new version's guesses with the old version's, for each group of users.12
At Big Widget: R13 added only the "region" input. But it also changed how much it trusted a dealer's order history. That is why brand-new dealers got such high forecasts. Comparing old and new guesses for dealers with little history would have shown it.
Loops you cannot see
Sometimes an AI's guesses change the world, and the changed world becomes its next lessons. When this runs through another system, it is hard to see. The paper calls these hidden feedback loops.1
Fix: for loops you can see, the paper suggests keeping some data outside the loop, so the AI can learn from things it did not cause.1 The paper gives no fix for hidden loops. This course uses the same idea for them.
At Big Widget: the forecast feeds the pricing tool. Where the forecast is high, discounts shrink. Smaller discounts mean fewer orders. Those orders become next month's lessons. So the forecast partly learns from its own effect. The course's fix: a group of dealers, picked at random, whose prices do not come from the forecast. Some of them teach the AI. The rest are kept only for checking it.
Users nobody told you about
Some people use the AI's output without telling its owners. The paper calls them undeclared consumers. When the AI changes, their work breaks, and nobody knows why.1
Fix: lock the output so only people on a list can read it, and tell them before each change.12
At Big Widget: the finance team had a spreadsheet that read the forecast log. Nobody told the forecast team. When R13 changed the forecasts, finance's numbers moved and nobody knew why.
4. Putting it together
Your plan for running one AI has four parts:
- The hold card from Level 300: the holder, the limits, the stop.
- The warning lights: the three kinds, which one wakes the holder, kept with the version.12
- The switch plan: one version number for all four parts (this course's rule), small steps, named people who can say stop, and a practised switch back.25
- The hidden-problem list: each tangle, loop and unknown user, with its fix.1
NIST says to keep applying this kind of care for as long as the system runs, as things change.5 The plan is never finished.
The three yeses, for a running AI
Module 100.9 taught the AiGovOps Foundation's three moves: Get to Yes, Stay at Yes and Recover to Yes. You get to yes, stay at yes and recover to yes. Here is what they mean for this plan. A plan is not permission. A named person says yes. The AI does not.
- Get to Yes, before each step of a switch. The check says pass or fail. Then Dana gives one of the four answers, and writes down what it was based on.
- Stay at Yes, while it runs. The warning lights keep checking. When the real answers come back, check them, even if no light came on. The first yes is not the last check.
- Recover to Yes, when a light goes off. Hold big orders. Switch back all four parts. Write down what the bad version touched. Fix the cause. Switching back is not the same as being fixed. You are back at yes only when the real answers, or another agreed check, show the old version is good enough for the groups that went wrong, and Dana agrees. Until then, the honest answer is "not known yet".
What to keep private. The team's own records keep the list of bad forecasts and the dealer details, locked so only the right people can read them. A receipt of a decision holds no private content. It says what was decided, by whom, when, which version, and points to the evidence. It never holds dealers' forecasts, orders or names. Here the receipt can be a plain, unsigned record. Modules 400.8 and 400.9 show how to make one a stranger can check.
Before you say yes
You should be able to say each of these out loud. "Not known yet" is an honest answer, as long as you also say who will find out.
- I know which warning wakes the holder, or I know that I do not.
- I know how often it would wake the holder for nothing, or I say "not known yet" and who will measure it.
- I can switch back all four parts, and we have practised it on a calm day.
- I know who reads the AI's output, and they know about the next change.
- I know who says yes at each step. It is a person, not the AI.
What this module does not claim
- The 2022 interviews talked with 18 engineers. What they found is a pattern in those talks, not a proven cause.4
- The 2019 study tested pictures, not forecasts. Its numbers may not fit other kinds of data.3
- The one-version-number rule, the "are the guesses strange?" light and the split group of dealers are this course's ideas, built on the papers. The papers did not test them.
- A good plan does not make an AI safe. It shows what you watch, who decides and how you switch back. Finishing this module says something about your practice, not about any system.
Further learning (not sources)
Free university course pages, opened on 18 Nov 2026. Nothing in this module rests on them: Stanford CS 329S Machine Learning Systems Design (https://stanford-cs329s.github.io/); University of Washington CSE 599W Systems for ML (https://dlsys.cs.washington.edu/); University of Michigan EECS 598 Systems for Generative AI, Winter 2024, public slides, not checked as complete (https://github.com/mosharaf/eecs598/tree/w24-genai).