
Capstone: publish work that survives review · Plain gear
Your last project
Figure data as text
Data and analysis
| Term | Data | Analysis | Source |
|---|---|---|---|
| Reproducible | Same | Same | [s3] |
| Replicable | Different | Same | [s3] |
| Robust | Same | Different (for example, reimplemented code) | [s3] |
| Generalisable | Different | Different | [s3] |
The lesson's terminology compares data and analysis: reproducible, replicable, robust and generalisable. The claim covers only the step actually repeated (s3).
This is the last project of Level 500. You make one piece of work that a stranger can check. Then other people check it, step by step, before anyone outside the course sees it.
You choose one of three kinds of work:
- Redo one result from a research paper. You try to get the same answer the paper got.
- Write the rules for one AI system at Big Widget Company. You say who decides, what must be checked first, what proof is kept and how someone can complain. You started this in 500.8.
- Attack one AI system on paper, with permission, and report what you found. You started this in 500.9.
In class you have 120 minutes. In that time you get your work ready and fill in a checklist. Other people's checks happen later, outside class.
Big Widget Company and its people are made up. The worksheet, the Foundation's web page and the research papers are real, and each one is cited.
Reviewed and published, version 1.0.0.
People make every choice here. An AI tool can help you write. But an AI tool never counts as one of the people who check your work, and it never signs a check. Sharing your work can't be undone. So it waits for two named people: Bob and Ken.
The idea behind it
The Foundation has a free study plan built from university courses. Its hardest stage ends with a project. You pick a question, try to redo one paper about it, write up "what held and what didn’t," and share it.1 Then you take it to the Foundation's Level 400 worksheet.1 The Foundation says no university made or approves this plan, and it gives no credit or certificate.1
The Level 400 worksheet is called "Prove it to a stranger."2 Its big idea is short. It says "Independence is about who made it, not how far away they are."2 In other words, the person who checks your work must not be you.
This course adds one rule. You may share your work only after every check has passed and Bob and Ken have said yes to that exact version. Until then it stays private, marked "draft."
Four answers
In this course, every check gives one of four answers:
- Go ahead. It passed, for this exact version.
- Only this far. You may claim a smaller thing, and no more.
- Wait for a person. Something is open. A named person has to act first.
- No. It did not pass. The work stays private. A "no" is final for that version: a later "go ahead" cannot undo it. To go on, you fix the work and label a new version.
"Wait" is not a pass. A wait names the person who must act. It has no end date. It never turns into a yes by itself, and it does not turn into a no by itself either. While it stands, nothing is shared. It ends only when that check writes a new answer. A check that just says "I looked" is not an answer at all.
This is a rule for this capstone only, and it is different on purpose. Module 500.R teaches a "wait" with a time limit, which turns into a no when time runs out. Under either rule, nothing is shared while the check is still open. A time limit would never let the work out early.
The difference is the way back. Here, the named person can still say "go ahead" later. Then the same version can carry on, and the checks after it are done again in order. With a time limit, the wait would become a no when time ran out. A no means that version does not pass. You would have to fix it, label a new version and take it through the checks again.
Think of a recipe
Say you claim you baked a great cake. Saying so proves nothing. A friend has to bake it in their own kitchen, from your written recipe, and get the same cake.
For that to work, your recipe must say:
- what you claim, and no more. "This cake rises well at sea level." Not "this is the best cake ever."
- what would prove you wrong. "If it comes out flat, my claim fails." Write this down before your friend bakes.
- where every ingredient came from, and which brand and size.
- steps a stranger can follow without calling you.
- what you are not sure about. "I have not tried it at high altitude." Say "not checked" or "I don't know" when that is the truth. Say who could find out.
That is the common rubric. Every route needs those five parts.
Researchers have written about why this matters. Two of them, Lipton and Steinhardt, said that papers should clearly separate a hunch from a result that evidence supports.3 Another team, led by Pineau, listed reasons that machine learning results fail to repeat. One reason is "Over-claiming": saying more than the evidence shows.4
Write the version on the jar
Now say you change the recipe. You add more sugar. Is it still the same cake? No. Your friend's test of the old recipe tells you nothing about the new one.
So you write a version on every recipe card. In this project you do the same. You give your work an exact version label that cannot quietly change. Every check then says which version it checked.
Here is a real case of one title with two versions. A paper called "Troubling Trends in Machine Learning Scholarship" came out online in 2018.3 A version with the same title came out in a magazine called ACM Queue in 2019.5 Even the authors' listed workplaces differ between the two.35 So when you cite it, you must say which one you read.
Here is another. At a big AI conference in 2019, authors filled in the same checklist twice: once when they sent their paper in, and again for the final version.4 That let the organisers see what had changed.4
Each check leaves a receipt
Each check writes a short note, like a shop receipt. It says which version, which check, who did it, the date and the answer. It may add a line about what the checker disagreed with.
A receipt holds nothing private. No personal details. Nothing from a private company or project. No copied chats or AI answers. It points to your files by their version label instead. Your working notes stay somewhere private.
Route 1: redo a paper
Pineau's team uses careful words.4 "Reproducible" means you used the same data and the same method and got the same result. If you change the data or the method, it has a different name.
Say exactly which step you redid. Did you only re-add up the paper's own answers? Then you checked the adding up. You did not check how the AI was built. Claim only the step you ran.
You pick one result from the paper. You compare it to a fair "baseline," which is the simple method you are measuring against. Lipton and Steinhardt describe a case where fancy new methods only looked better because the old method had not been tuned well. Once it was tuned, the plain old method came out on top.3 So make your baseline fair, or say you don't know how it was tuned.
Before you run anything, write down two things. First, which number from the paper you will compare against. It must be for the same data you use. Second, how close counts as a match.
You also say how sure you are. If your results change from run to run, run it more than once. Report the average and how much the results moved around. Pineau's checklist asks for that kind of report.4 If your runs all give the exact same number, say so, and say how many runs you did. Don't claim it can never change. A script can use chance, or something online, or a setup you did not lock down.
Do not expect a simple yes or no. Pineau's team says there is "rarely such a concise outcome" when people try to redo a study.4 Write down what matched and what did not.
When you cannot redo the whole paper
Sometimes you just cannot. The data may be private. Or the work may need so many computers that most researchers could not try.4 For private data, Pineau's team describes one fix: also show results on a public dataset.4
You can do something like that. Pick a smaller claim that you can check. That is an "only this far" answer. For example, check the result on a small public slice of the data that the authors froze and shared. Write down exactly what went in, what should come out and how to run it. The paper must give a number for that same slice. If it only gives a number for the whole test, you can't compare to it.
Then say only what you really checked. A slice of inputs on its own has no right answer to match. Maybe the run worked and gave a score. Maybe a second run gave the same score. Or maybe the authors also shared the answers or scores for those exact inputs, and your run agreed with them. Then name that file. Each of these is a different, smaller claim. None of them says the paper's result held.
Then someone else must really run it. They write down what they got.
What if there is no smaller check at all? Then you write down the problem honestly. But your work stays private and draft. Writing "I couldn't get the data" is honest. It is not the same as someone checking your work. It never passes the "someone else ran it" step.
Route 2: write the rules
This builds on 500.8. You hand in:
- who uses the system, for what, and from what date;
- what you assume about the law, marked as assumptions;
- a table: each rule, the check before the action, the proof kept, and the person who owns it;
- a "stops when proof is missing" test. Make a labelled practice copy of one piece of proof. Take the proof out of the copy. Show that the action stops. Never touch the real file;
- a "follow one decision" test. Someone else uses your real, complete files to trace one decision from rule to check to proof to owner;
- a walk-through: one person complains, and you show what happens next.
The two tests check different things. The first shows that missing proof stops the action. The second shows that someone else can rebuild a decision from what you kept.
A research team led by Raji designed a way for companies to check their own AI. Each stage of their check makes documents, and together those documents form the final report.6 Their record of the work is meant to let someone rebuild the key decisions later.6 Your "follow one decision" test is where a stranger tries that for one decision.
Do not say a system is "certified." Naming a standard is not the same as passing it.
Route 3: attack it, with permission
This builds on 500.9. You hand in:
- written permission to do the test;
- tests on made-up conversations that the course owns, so anyone can run them again;
- a log of what you tested, what you did not, and when you stopped;
- a ruling on each thing you found, made by someone else;
- the fix, and a test showing whether the fix worked;
- what is still not covered.
You cannot judge your own finding. Another named person does. Raji's team explains why. People who check their own company's work share that company's interests.6
The checklist you finish in class
Before you hand anything over, tick every box:
- I chose one route.
- I gave my work and all my proof an exact version label.
- I wrote one small, clear claim. It names the step I really redid.
- I wrote what would prove me wrong, before anyone ran it.
- I listed where every input came from.
- I wrote steps a stranger can follow.
- I listed what is missing and how unsure I am.
- (Paper route) I have a smaller check someone else can run, or I wrote "none, so it stays private."
- Nothing private is in it: no personal details, and nothing from a private company or project.
- It is marked private and draft.
The Level 400 worksheet asks the same about privacy. When you check someone's bundle there, it fails if it holds a prompt, an AI answer or personal data.2
Six checks, in order
After class, your work goes through six checks. They go in this order. Each one writes down the exact version it checked. A check counts only if its answer is "go ahead."
- Someone else reruns it. Not you. They follow your steps and write down what they got. It passes only if the run worked and matched what you said would come out. If a step is missing, it does not pass. You fix it, label a new version, and they run it again.
- Someone attacks it. Not you. A different named person rules on each thing they find.
- Fix and recheck. You fix what was found. Others test the fix and check your sources again. Every problem that was ruled right must be fixed and retested before step 4.
- Two people review it. One knows the subject well. One is the kind of reader it is for. Both must say yes. A computer grade does not count.
- A privacy check. A named person makes sure nothing private would go public.
- Bob and Ken decide. They write down a yes or no for that exact version.
No step may happen before the one above it. If someone disagrees, their note stays in the record. A disagreement is never counted as a yes.
Only the newest answer at each check counts. Say the privacy checker says "wait" after Bob and Ken said yes. Now the old "go ahead" no longer counts, and nothing is shared. If the privacy checker later says "go ahead" again, Bob and Ken must decide again, because their yes came first.
A "no" works differently from a "wait". Once any check says "no" to a version, that version can never be shared, even if a "go ahead" comes later. You fix the work, label a new version and take it through the checks again.
How do you know which answer is newest? Each answer goes into one log, and the log gives it the next number: 1, 2, 3 and so on. The higher number is newer. Clock times can match, so they do not decide it. If a check has no answer yet, or you cannot tell which answer came last, the work is not ready to share.
Why must the checker be someone else? On the Level 400 worksheet, you check a stranger's work by following their written steps on your own, with no Foundation tool and no account.2 A big AI conference ran a challenge like this in 2019. People from 73 places tried to redo results from 173 papers.4 The goal was checking by people who did not do the original work.4
As of November 2026, the worksheet marks some of its tools as working today and others as coming later.2 Use only what works today. This project needs no tool and no account.
When your work changes
Go back to the cake. Your friend tested version 1. Then a reviewer says, "the sugar is too low." You change the recipe. Now it is version 2.
Your friend's test was of version 1. It does not count for version 2 by itself. Neither does anything else done on version 1: not the attack, not the reviews, not the privacy check, not Bob and Ken's yes.
So when you change the claim, the comparison, the data, the code or the proof, you:
- label the new version;
- have the rerun and the attack ruling done again, if the change touches them;
- fix and recheck what was found, and check the sources again;
- get both reviewers to look at the new version;
- get the privacy check again;
- then ask Bob and Ken.
What if a change does not touch an earlier check? Say you only cut a sentence that claimed too much. Even so, nothing carries over by itself. The person who did that check looks at what changed. They ask two questions:
- Did the change touch what I ran?
- Does my result still back the new claim?
The second question matters. New words can change what counts, even when the same job ran. A claim about a different group, or a different score, or a different "close enough," needs a new check.
If both answers are safe, they write a new note for the new version: "This change does not touch what I checked, because …" The note gives both reasons. Their old note is the proof behind the new one. Only they can write it, never you. The old note must be a "go ahead." If it said "no" or "wait," it cannot back the new note. The check has to be done again.
The reviews, the privacy check and Bob and Ken's yes are always done again for the new version.
Until all six are done for the new version, it stays private and draft.
A story: Ana and the frozen slice
Ana works in research at Big Widget Company. (She and the company are made up. So is the paper in this story.)
Ana wants to check one result from a paper. Building the AI and the full test need far more computer power than she has. But the authors shared their finished AI, a simpler AI to compare against, and a small, frozen slice of their test data. The paper also lists each AI's score on that exact slice.
So Ana makes a smaller claim. "On this shared slice, running these two shared AIs gives the scores the paper lists, as close as I said in advance. The new AI scores higher. I checked running and scoring only. I make no claim about how they were built, or about the full test." Before any run, she writes down how close counts as a match. She writes what would prove her wrong. She lists every file with its version label. She writes steps for a stranger. She calls this version 1.
Ben, a coworker, runs her steps on version 1. It fails: one step is missing. His answer for version 1 is "no." Ana fixes the steps. That is a change, so it is now version 2. Ben runs version 2. Both scores match. He runs it twice and gets the exact same numbers. He writes that down as what he saw in two runs, not as proof it can never change. He writes "go ahead" for version 2.
Then a red-teamer reads the paper closely. The simpler AI got much less tuning than the new one. So "the new AI scores higher" may be about tuning, not about the new idea. A different person rules that the finding is right.
Ana can't retune it. So she makes her claim smaller. She adds: "The two AIs were tuned differently, so this does not show why one scored higher." That changes her claim, so it is version 3. Ben looks at what changed. Nothing he ran changed. The new sentence only takes away a guess about why. The slice, the scores and "how close counts" are all the same. So his run still backs the smaller claim. He writes a new note for version 3: "not touched," with both reasons. It points to his "go ahead" for version 2, not his "no" for version 1. The finding is checked again on version 3. Someone else checks her changed sentence against the sources.
Two reviewers read version 3, after the fix. Privacy is cleared for version 3. Bob and Ken say yes to version 3. Only version 3 is shared. Versions 1 and 2 never are.
What if the authors had shared nothing? Then Ana would write that down honestly. Her work would stay private and draft.
What passing means
If your work is shared, it means one small claim, at one version, held up when other people checked it. It is proof of that, not a certificate. It does not mean the whole paper is right. It does not mean a system is safe or legal.
It is not a degree. The Foundation says its study plans "are not degrees."1 The Level 400 worksheet says that systems get scores "from their own evidence," not from a person's level.2
Three-yes check
- Get to Yes: before any check, a small claim, a way to be wrong written in advance, a list of inputs, steps for a stranger, and honest gaps, all with one version label.
- Stay at Yes: every check names the same version, says "go ahead," and is done in order, by someone else where it must be. No receipt holds anything private.
- Recover to Yes: a "wait" names who must act, and the same version can go on once they say "go ahead". A "no," or any change, means a new version. Name who will fix it. Label the new version, get a fresh check or a new "not touched" note for each early check, redo the reviews, the privacy check and Bob and Ken's yes. Saying "fixed" is not enough until the checks pass again.