Level 400: Engineering AI systems · Module 400.4

At a café counter, an engineer raises a hand to pause while a colleague holds a plain clipboard.
Who can stop it?

Operating a model: monitoring, drift and rollback · Plain gear

As of 18 Nov 2026. Reviewed and published, version 1.0.0, part of How AI Works, the AiGovOps Foundation's planned free course. It is not a credential. The research comes from dated papers, and it is true of what those papers studied. There is no real incident here. Big Widget's story is made up, and says so. If we do not know something, we say so.

You will be able to:

Big Widget Company is made up for this course. So is everyone and every number in its story.

The story: the forecast and the new region

Big Widget has an AI that guesses how many widgets each dealer will order next week. This guess is called the forecast. The parts team buys parts based on it. The pricing tool reads it too.

Here is the catch. A dealer's orders are only final six weeks later. So Big Widget learns whether a forecast was right six weeks after it was used. The right answer is called the label. Here, labels come six weeks late.

One autumn, Big Widget switched to a new version of the forecast. Two weeks later, a new sales region opened: Region 9, with 40 new dealers. The rest of this module follows what happened.

1. Warning lights that do not wait for the answers

The mistake this fixes: "If the AI passed its tests, it stays good."

The tests were run on old data. The world keeps changing. Engineers at Google wrote in 2015 that tests alone cannot show a system is still working "in the face of a changing world". They said live watching, with an automatic response, is key to keeping a system reliable.1 A 2017 paper from Google said it plainly: you need to know the system "continues to work correctly over time".2

The trouble is often silent. A 2019 study said AI systems "tend to fail silently" when their inputs change. It also said that, in practice, AI systems rarely check incoming data for this kind of change.3

Think of a car dashboard

A car does not wait until the engine breaks to warn you. It has lights for low oil, hot engine and low tyre pressure. Each light is an early sign.

An AI system needs early signs too, because the real answers (labels) come late. In a 2022 study, researchers talked with 18 engineers who run AI systems. Many said labels often came late, by an amount that was hard to predict. One said feedback was "always delayed by at least 2 weeks".4 This study was not checked by other experts before it was posted.

Three kinds of warning light

  1. Are the inputs strange? Check that today's data looks like the data the AI learned from. Are there new kinds of dealer? Missing values? Numbers out of range?23
  2. Are the guesses strange? Compare the AI's guesses with a "normal" you pick: its own guesses when it was launched, or today's guesses for similar dealers. Do it for each group of dealers. This check is this course's own idea of an early warning. It does not need the real answers, so it cannot tell you the guesses are right. Also set limits on what the AI is allowed to do. If it hits a limit, someone looks.1
  3. Were the guesses right? When the labels finally arrive, compare the guesses with the real answers, group by group.1 Check that the quality is still as good as it was at launch.2

Lights 1 and 2 come early. Light 3 is the proof, but it comes late. Light 2 only tells you the guesses have moved. Two sets of guesses can look the same today and still turn out very differently.

Two warnings about warning lights. First, a change in the data is not always harm. The 2019 study notes that data changes all the time, and often the change does no damage.3 A light means "look now". It does not mean "the AI is wrong". Second, too many lights is as bad as none. In the 2022 interviews, false alarms were the most talked-about problem. People got so many that they started to ignore them, and so could miss real drops.4

Start from your hold card

At Level 300 you made a hold card. It names one person, the holder, who is responsible for the system. It lists the limits that matter. And it has a stop that makes the system wait safely if something goes wrong. Keep the card's label saying it was made at Level 300.

Now add the warning lights to that card. This course's rule: for each kind of light, pick one warning that wakes the holder (a "page"). Everything else goes on a to-do list. A warning level is a choice, and a person owns it. Before you trust it, try it on past days when nothing went wrong, and count how often it would have woken the holder for nothing. If nobody has done that yet, the honest answer is "not known yet". Say so, and name the person who will find out.

The Google engineers warned that waiting for a person to answer a page can be too slow for urgent problems.1 That is why the stop acts on its own. The AI never places a big order by itself. Anything that cannot be undone waits for a person.

A warning light never decides anything. It tells the holder. The holder decides.

A US standards body, NIST, says the same kind of thing in its 2023 AI risk framework: once a system is running, there should be plans for watching it, for overriding it, and for responding when things go wrong.5

Keep the lights with the version

Each warning light compares today with "normal". But "normal" belongs to one version of the AI. If you change the version, change the lights' idea of normal too. If you switch back, switch the lights back. Google's engineers said settings like these should be reviewed and saved like code.1

What happened at Big Widget (made up)

The hold card names Dana Reyes, the demand-planning lead, as holder. The warning that pages Dana:

Dana picked these levels. They are made up for the story, not advice.

Region 9 opened on Day 15. Its dealers were a new kind the AI had never seen. They had almost no order history. On Day 16 the input check failed for the second day in a row. Region 9's forecasts were 2.6 times those of similar dealers. Dana was paged on Day 16. The labels would not have shown the problem until Day 57 or later.

On Day 16, Dana knew the inputs were new and the guesses were odd. Dana did not know yet if the guesses were wrong. That would be known from Day 57, and finding out was Dana's job. The page was a reason to look, and to hold big orders. It was not proof.

2. A plan for switching, and for switching back

Pin four things for every release: the model, the data, the code and the configuration. The course gives the four one release ID, and a rollback restores a release ID, never one part.
Know the exact code that produced a model (s2); version copies of input signals (s1); training and serving code can drift apart (s2); configuration is reviewed and checked in (s1). One release ID for all four is the course's rule.
Figure data as text

Four parts, one release ID

PartWhat it shows
MODEL [s2]Exact weights and spec. Know the exact code that was run to produce a given model.
DATA [s1]Training snapshot and signals. Keep versioned copies of input signals another team owns.
CODE [s2]Feature pipeline and serving. A newer model can need code an older server lacks.
CONFIGURATION [s1]Thresholds, features, monitors. Reviewed and checked into a repository.
ONE ID (COURSE)One release ID. A rollback restores a release ID, never one part.

The mistake this fixes: "Switching back an AI is like undoing a change to a program."

Think of a recipe

A cake depends on the recipe, the ingredients, the oven and the oven setting. Say a new cake comes out badly. You go back to the old recipe, but keep the new flour and the new oven setting. You still get a bad cake, or a different bad cake.

An AI is the same. Its behaviour depends on four things:

  1. The model: the trained AI itself.
  2. The data it learned from.
  3. The code that prepares its inputs and serves its answers.
  4. The settings: its limits and its warning lights.

The 2015 Google paper calls this "Changing Anything Changes Everything".1 The 2017 paper says being able to go back quickly to "a previous known-good state" is crucial.2 This course turns those two ideas into one rule: keep the four parts together under one version number. When you switch back, switch back all four. The rule is ours, built on the papers; the papers do not say it in these words.

The recipe is only a picture. Going back to the old recipe helps only if the old recipe suits today's kitchen. If something new has turned up, the old version has not seen it either.

Switch a little at a time

Do not give the new version to everyone on day one. Google's 2017 paper says to turn it on for a small share first, run the old and new side by side, and grow the share only while the new one behaves.2 Several companies in the 2022 interviews did this in stages, usually one to four. One stage is "shadow": the new version makes guesses, but nobody sees them.4

Be clear about what each step can show before the real answers come back. A shadow step shows the new version runs and its guesses look sensible. It cannot show what happens when people act on the guesses. A small share limits the damage if the new version is bad. Neither one shows the guesses are right. At Big Widget every step was over long before the first real answer came back.

So moving to the next step is a choice by a named person, never by the AI. There are four answers: go ahead (allow), only this far (constrain; say, 10% of dealers), wait for a person (hold; for example, until the real answers come back) or no (deny). Write down the answer, who gave it and what it was based on. "No warning yet" does not mean "the guesses are good".

Name who can say stop, and practise

The plan names the people who can stop the switch, and who can order a switch back. NIST says roles for switching off or replacing an AI system that is not doing its job should be given out and understood.5

Then practise switching back on a calm day. The 2017 paper says switching back is an emergency step, so people should practise it "when not in emergency conditions".2

Switching back fixes the system. It does not undo the bad guesses it already made. Write down which guesses the bad version made, so they are not used to teach the next version by mistake.

What happened at Big Widget (made up)

The new version, called R13, had a new model and new input code. The input code added a "region" field. On Day 14, Dana said yes to R13 for all dealers. No real answers had come back yet, so that yes rested on the early checks. Region 9 opened the next day, so the steps never tested it.

On Day 16, the engineer on duty switched back only the model to the old one, from R12. It did not work. The new input code was still sending the "region" field. The old model did not expect it. The forecasts stopped, and the stop on the hold card held big parts orders for a person to check.

Then the engineer switched back all four parts to R12: the old model, the old data record, the old code and the old settings, warning lights included. The forecasts came back. The team wrote down every Region 9 forecast R13 had made.

The old version had never seen Region 9 either. So Dana kept the hold: big Region 9 parts orders still waited for a planner to check them. A hold is not forever. Dana wrote down when to look at it again: when Region 9's real answers start to come back, from Day 57.

The lesson: a switch-back plan that saves the old model but not the old input code is not a switch-back plan.

3. Hidden problems: the AI is the small part

The mistake this fixes: "The AI model is most of the system."

The 2015 Google paper drew a picture. The AI code was a small box in the middle. Around it were big boxes for collecting data, preparing it, serving answers, settings and watching. They wrote that only "a small fraction" of real systems is the AI code.1 They also said building an AI system is fairly quick and cheap, but keeping it running is hard and costly, and hidden problems grow quietly.1

Here are three hidden problems, each with one fix.

Tangled inputs

The AI mixes its inputs together. Change one input, and the AI may change how much it uses all the others. The paper says no inputs "are ever really independent".1

Fix: before switching, compare the new version's guesses with the old version's, for each group of users.12

At Big Widget: R13 added only the "region" input. But it also changed how much it trusted a dealer's order history. That is why brand-new dealers got such high forecasts. Comparing old and new guesses for dealers with little history would have shown it.

Loops you cannot see

Sometimes an AI's guesses change the world, and the changed world becomes its next lessons. When this runs through another system, it is hard to see. The paper calls these hidden feedback loops.1

Fix: for loops you can see, the paper suggests keeping some data outside the loop, so the AI can learn from things it did not cause.1 The paper gives no fix for hidden loops. This course uses the same idea for them.

At Big Widget: the forecast feeds the pricing tool. Where the forecast is high, discounts shrink. Smaller discounts mean fewer orders. Those orders become next month's lessons. So the forecast partly learns from its own effect. The course's fix: a group of dealers, picked at random, whose prices do not come from the forecast. Some of them teach the AI. The rest are kept only for checking it.

Users nobody told you about

Some people use the AI's output without telling its owners. The paper calls them undeclared consumers. When the AI changes, their work breaks, and nobody knows why.1

Fix: lock the output so only people on a list can read it, and tell them before each change.12

At Big Widget: the finance team had a spreadsheet that read the forecast log. Nobody told the forecast team. When R13 changed the forecasts, finance's numbers moved and nobody knew why.

4. Putting it together

Your plan for running one AI has four parts:

  1. The hold card from Level 300: the holder, the limits, the stop.
  2. The warning lights: the three kinds, which one wakes the holder, kept with the version.12
  3. The switch plan: one version number for all four parts (this course's rule), small steps, named people who can say stop, and a practised switch back.25
  4. The hidden-problem list: each tangle, loop and unknown user, with its fix.1

NIST says to keep applying this kind of care for as long as the system runs, as things change.5 The plan is never finished.

The three yeses, for a running AI

Module 100.9 taught the AiGovOps Foundation's three moves: Get to Yes, Stay at Yes and Recover to Yes. You get to yes, stay at yes and recover to yes. Here is what they mean for this plan. A plan is not permission. A named person says yes. The AI does not.

What to keep private. The team's own records keep the list of bad forecasts and the dealer details, locked so only the right people can read them. A receipt of a decision holds no private content. It says what was decided, by whom, when, which version, and points to the evidence. It never holds dealers' forecasts, orders or names. Here the receipt can be a plain, unsigned record. Modules 400.8 and 400.9 show how to make one a stranger can check.

Before you say yes

You should be able to say each of these out loud. "Not known yet" is an honest answer, as long as you also say who will find out.

What this module does not claim

Further learning (not sources)

Free university course pages, opened on 18 Nov 2026. Nothing in this module rests on them: Stanford CS 329S Machine Learning Systems Design (https://stanford-cs329s.github.io/); University of Washington CSE 599W Systems for ML (https://dlsys.cs.washington.edu/); University of Michigan EECS 598 Systems for Generative AI, Winter 2024, public slides, not checked as complete (https://github.com/mosharaf/eecs598/tree/w24-genai).

Sources

  1. s1 D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, Dan Dennison (2015). Hidden Technical Debt in Machine Learning Systems. Paper. Checked 2026-11-19 by Source checker (Claude, round-3 re-check session, separate from the 400.4 writer, reviser and earlier checkers; full NeurIPS PDF, Figure 1 rendered from vector paths). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  2. s2 Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib, and D. Sculley (2017). The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. Paper. Checked 2026-11-19 by Source checker (Claude, round-3 re-check session, separate from the 400.4 writer, reviser and earlier checkers; full PDF from the research.google record). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  3. s3 Stephan Rabanser, Stephan Günnemann, and Zachary C. Lipton (2019). Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift. Paper. Checked 2026-11-19 by Source checker (Claude, round-3 re-check session, separate from the 400.4 writer, reviser and earlier checkers; full arXiv v4 PDF). ↩ ↩ ↩ ↩
  4. s4 Shreya Shankar, Rolando Garcia, Joseph M. Hellerstein, and Aditya G. Parameswaran (2022). Operationalizing Machine Learning: An Interview Study. Paper. Checked 2026-11-19 by Source checker (Claude, round-3 re-check session, separate from the 400.4 writer, reviser and earlier checkers; full arXiv v1 PDF). ↩ ↩ ↩ ↩
  5. s5 Elham Tabassi (National Institute of Standards and Technology) (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). Standard. Checked 2026-11-19 by Source checker (Claude, round-3 re-check session, separate from the 400.4 writer, reviser and earlier checkers; full NIST AI 100-1 PDF). ↩ ↩ ↩ ↩

Next: the quiz