
Capstone: publish work that survives review · Technical gear
Deliverable and scope
Figure data as text
Data and analysis
| Term | Data | Analysis | Source |
|---|---|---|---|
| Reproducible | Same | Same | [s3] |
| Replicable | Different | Same | [s3] |
| Robust | Same | Different (for example, reimplemented code) | [s3] |
| Generalisable | Different | Different | [s3] |
The lesson's terminology compares data and analysis: reproducible, replicable, robust and generalisable. The claim covers only the step actually repeated (s3).
The Level 500 capstone is one version-pinned artifact on one of three routes, plus an ordered set of version-bound gate records produced by people other than the author.
| Route | Builds on | Artifact |
|---|---|---|
| Paper reproduction | 500.1 | A bounded reproduction of one result, or of a narrower claim via a stranger-runnable substitute |
| Governance regime | 500.8 | A dated regime for one Big Widget Company system with a clause → gate → evidence → owner table |
| Red-team report | 500.9 | An authorised exercise on course-owned synthetic Big Widget transcripts, with independent rulings |
The in-class allocation is 120 minutes. It covers preparing the artifact from supplied data or a carried-forward draft, pinning its version and completing the pre-submission evidence and privacy checklist. Full replication, external review and release are out-of-class steps.
Big Widget Company and its staff are fictional. Every external fact below is cited. The release rule (publish only the version for which every gate passes and Bob and Ken record a release decision) is this course's rule for Level 500, not a property of any cited source. No agent counts as an independent rerunner, red-team ruler, reviewer, privacy clearer or release decider, and no agent signs a gate record. Publication is irreversible, so it waits for two named people.
Reviewed and published, version 1.0.0.
Provenance of the capstone
The capstone adapts the PhD-level capstone in the Foundation's free university-course programme: "Pick one open question from the seminars." "Reproduce one paper on it, write up what held and what didn’t, and publish it." "Then put your result through Practice Level 400: prove it."1 The page states "The order, the levels and the capstones are the Foundation’s suggestion." and "No university designed or endorses these programs, and none of them gives credit or a certificate through us."1
Practice Level 400, "Prove it to a stranger", is a 60-minute worksheet.2 Its stated premise is that "the audit gate is about independence, not skill", and "Independence is about who made it, not how far away they are."2
Two deliberate differences from the Foundation's version:
- The Foundation's capstone ends in "publish it".1 This capstone ends in a version-bound release decision, and the artifact is private and draft until then.
- The Foundation's capstone is a paper reproduction.1 This capstone admits two further routes, governance regime and red-team report, under the same common rubric.
Common rubric
Every route must state:
| Field | Requirement |
|---|---|
claim | One bounded claim: population or data, conditions, metric, version, and the pipeline stage actually re-executed |
falsifier | The observation, or failure condition, under which the claim is rejected; pinned in $V$ before any rerun |
provenance | Every input with source and immutable version identifier |
stranger_check | Instructions that a person with no contact with the author can execute, with expected outputs and the comparison rule |
limitations | Missing evidence, unchecked items, and quantified uncertainty where a quantity is claimed; each item labelled not checked, could not open or unknown, with an owner and next check |
Lipton and Steinhardt's desiderata map directly onto claim and falsifier: papers should "provide intuition to aid the reader’s understanding, but clearly distinguish it from stronger conclusions supported by evidence" and "describe empirical investigations that consider and rule out alternative hypotheses".3 They name four trends: "Failure to distinguish between explanation and speculation."; failure to identify the sources of empirical gains; "Mathiness"; and misuse of language.3 Their suggestion to authors is to ask "what worked?" and "why?", rather than just "how well?", and they name "error analysis, ablation studies, and robustness checks" as practices common in the strongest empirical papers.3 They also recommend "being clear about which problems are open and which are solved".3
Pineau et al. list causes of the reproducibility gap in machine learning, including "Under-specification of the metrics used to report results", "Improper use of statistics to analyze results", "Selective reporting of results and ignoring the danger of adaptive overfitting" and "Over-claiming of the results, by drawing conclusions that go beyond the evidence presented".4 The rubric is aimed at each.
Version binding
The version object
Let the artifact files be $A = \{a_1, \dots, a_m\}$ and the evidence files $E = \{e_1, \dots, e_n\}$ (data, code, environment specification, logs, transcripts, outputs, comparison rule). A version is pinned either by immutable revision identifiers or by a file-hash manifest
V = \{(\text{path}(f), h(f)) : f \in A \cup E\},
where $h$ is a cryptographic hash. Two versions are equal only if their manifests are equal; any edit to any file in $A \cup E$, including the stranger-check instructions, yields $V' \neq V$. Because the comparison rule $\rho$ and the falsifier are files in $V$, a rerun on $V$ is by construction judged against a rule fixed before it ran.
The environment is part of $E$. Pineau et al. note that software containers "can encapsulate operating systems components, code and data into a single package", and that standardising such tools "would help sharing of information and improve ease of reproducibility".4 A container image digest or an exact dependency lock file are two ways to put the environment in the manifest.
Gate records
A gate record is a tuple
g = (k, V, \text{actor}, n, t, \text{answer}, D, \text{ref}),
with gate index $k$, the version $V$ it names, a named human actor, a sequence number $n$, a timestamp $t$, an answer in $\{\text{allow}, \text{constrain}, \text{hold}, \text{deny}\}$ (go ahead / only this far / wait for a person / no), a set $D$ of recorded disagreements, and $\text{ref}$, hashes of the evidence it relied on. A record covers only the $V$ it names. It is a receipt: it holds identifiers, hashes, the decision and short disagreement statements, and no private content (no personal data, private-repository text, or pasted prompts, outputs or transcripts). Working notes stay in a separately controlled record.
The record for a gate passes only on a favourable answer. Mere existence of a record is not a pass:
\text{ok}_k(g, V) \iff g.k = k \;\land\; g.V = V \;\land\; g.\text{actor} \in \text{Allowed}_k \;\land\; g.\text{answer} = \text{allow}.
A hold names the actor who can release it. It has no automatic expiry and no automatic transition: it does not turn into deny with time, and no sequence of holds yields allow. While a hold is the record in force for its gate (defined below), $\text{ok}_k$ is false and release is blocked. The hold ends only when an actor in $\text{Allowed}_k$ records a later answer for the same gate and version.
This no-expiry rule is a deliberate capstone exception, not the level's general hold policy. Module 500.R has the operational hold record carry a lifetime after which it becomes a no. Under either policy an unresolved hold blocks release: hold and deny both make $\text{ok}_k$, and so $\text{Release}$, false, so an expiry would never permit publication while the gate is unresolved.
The policies differ in the path back to release. Here, an actor in $\text{Allowed}_k$ may later record allow on the same $V$; that record becomes $\text{In}_k$, and the ordering clause below then requires a fresh record at every gate $j > k$. Under an expiry, the hold would become a deny on $V$. In this course a deny is final for $V$: once any gate records deny on $V$, no later record on $V$, at that gate or any other, can make $\text{Release}(V, \cdot)$ true (the $\text{Denied}$ clause below). The only route back is to fix the work and pin $V'$, which goes through "Material change and reopening": impact records where eligible, or fresh records, for gates 1–3, and fresh records for gates 4–6. So the same $V$ can still publish here once its hold is released, while the expiry route leads to a new version.
Disagreements are evidence, not approvals. Each $d \in D$ stays in the record. If any $d$ undermines the claim or prevented the pinned procedure from running, the actor's answer must be deny or hold, never allow.
The release predicate
Let $\mathcal{R}_k(V)$ be the records for gate $k$ that name $V$ and were made by an actor in $\text{Allowed}_k$ (impact records are added in "Material change and reopening" below). Records are never edited or deleted; a later record supersedes an earlier one.
"Later" is defined by position, not by clock. All gate records, finding acceptances and retest closures for $V$ go into one append-only log, which assigns each entry the next sequence number $n$. A well-formed log has exactly one entry at each position $1, \dots, N$: no gaps, no repeats, and no entry without a number. Timestamps $t$ are kept for people to read; they may be rounded or equal, and they never decide supersession. The record in force for gate $k$ on $V$ when the log stands at position $N$ is
\text{In}_k(V, N) = \arg\max_{g \in \mathcal{R}_k(V),\; n(g) \le N} n(g),
which is unique when the log is well formed, because no two entries share an $n$. If no record in $\mathcal{R}_k(V)$ has $n(g) \le N$, then $\text{In}_k(V, N)$ is undefined and $\text{ok}_k$ is false. If the log is not well formed, the record in force cannot be identified, and release is denied rather than resolved by guessing; in particular, an allow and a hold that share a timestamp are ordered by $n$, never by a choice between them.
A later record for the same gate and version supersedes an earlier one whatever its answer: a privacy hold recorded after a privacy allow is the record in force, and the allow no longer counts. A deny is the exception: it is never lifted on the same version. Define
\text{Denied}_k(V, N) \iff \exists\, g \in \mathcal{R}_k(V) :\; n(g) \le N \;\land\; g.\text{answer} = \text{deny},
\qquad
\text{Denied}(V, N) \iff \bigvee_{k=1}^{6} \text{Denied}_k(V, N).
Publication of $V$ when the log stands at position $N$ is permitted only if
\text{Release}(V, N) \iff \text{WF}(V, N)
\;\land\; \neg\,\text{Denied}(V, N)
\;\land\; \bigwedge_{k=1}^{6} \text{ok}_k\big(\text{In}_k(V, N), V\big)
\;\land\; n\big(\text{In}_1(V,N)\big) < \dots < n\big(\text{In}_6(V,N)\big) \le N
\;\land\; \text{OpenAccepted}\big(V, n(\text{In}_4(V,N))\big) = \varnothing
\;\land\; \text{OpenAccepted}(V, N) = \varnothing ,
where $\text{WF}(V, N)$ says the log for $V$ is well formed up to $N$, every $\text{In}_k(V, N)$ is defined, and $\text{OpenAccepted}(V, m)$ is the set of findings accepted against $V$ at positions up to $m$ that are not closed with a passing retest by position $m$. The predicate is evaluated against the log as it stands at the moment of publication, not at the moment the six allows first existed. So:
- a hold or deny recorded at any gate for $V$ after the six allows, and before publication, becomes the record in force for that gate and makes $\text{Release}(V, N)$ false;
- if that gate held and later records allow again, its in-force record now postdates the gates after it, so the ordering clause fails until every later gate records a fresh decision; in particular Bob and Ken must decide again after a re-cleared privacy gate;
- if that gate denied, $\text{Denied}(V, N)$ stays true at every later $N$, so no later allow on $V$ restores release. Paper trace: allows at positions 1–6, a gate-5 deny at 7, a gate-5 allow at 8 and a gate-6 allow at 9 leave positions 1, 2, 3, 4, 8 and 9 in force and correctly ordered, but the deny at 7 makes $\text{Release}(V, 9)$ false; the author pins $V'$;
- a finding accepted against $V$ after review blocks release until it is closed with a passing retest, and a new gate-2 ruling or gate-3 closure after review breaks the ordering, so review is redone.
The ordering clause enforces gate order; the first $\text{OpenAccepted}$ clause enforces that formal review starts only after every accepted finding is closed. $\text{Allowed}_k$ excludes the author where independence is required (table below). There is no inheritance rule: $\text{ok}_k(g, V) \not\Rightarrow \text{ok}_k(g, V')$.
Two real examples of version ambiguity
Preprint versus publication. "Troubling Trends in Machine Learning Scholarship" exists as arXiv 1807.03341v2, stamped 26 July 2018, which states it answers "a call for papers from the ICML Machine Learning Debates workshop".3 Crossref holds a separate record: ACM Queue volume 17, issue 1, pages 45–77, issued February 2019, publisher "Association for Computing Machinery (ACM)".5 Metadata differ between the two: the arXiv PDF lists the affiliations "Carnegie Mellon University, Stanford University", whereas the Crossref record gives Steinhardt's affiliation as "UC Berkeley".53 This module read the 2018 full text and only the 2019 metadata; it quotes only the former and does not transfer the Queue venue to the preprint, or preprint content to the Queue article.
Checklist answers bound to a paper version. At NeurIPS 2019, checklist version 1.2 was completed "both at the initial paper submission phase (May 2019), and at the final camera-ready phase (October 2019)", which "allowed us to analyze any change in answers".4 The answers attach to a paper version, not to the paper in the abstract.
Route 1: paper reproduction
Terminology
Pineau et al. adopt the following definitions.4
| Same analysis | Different analysis | |
|---|---|---|
| Same data | Reproducible | Robust |
| Different data | Replicable | Generalisable |
They define reproducibility as "obtaining similar results as presented in a paper or talk, using the same code and data (when available)", and call it "a necessary step to verify the reliability of research findings."4 Label your route-1 artifact with the cell it occupies, and with the pipeline stage re-executed: training, inference, or scoring of released predictions. Re-scoring released predictions verifies the scorer and the reported number for those predictions; it does not verify inference or training. The claim covers only the stage that ran.
Required content
For the pinned $V$:
- Pin the paper version, data, code and environment.
- Reproduce one bounded result against a stated baseline.
- Pin a comparison rule $\rho$ before any run: the reference value $r$, its location in the paper, and the acceptance condition (for example a tolerance and the reason for it).
- Report mismatches, and uncertainty in your own estimate.
The reference must be matched: $r$ must be reported for the same population (the same subset or released artifact) and the same statistic as your run. A full-benchmark headline is not a reference for a subset. Without a matched $r$, narrow the claim to exactly what was checked, and say so. A frozen input subset $I_S$ has no expected output of its own, so three findings must be kept apart:
- completion: the pinned procedure ran on $I_S$ and produced $y_S$;
- observed repeatability: $j$ reruns on $I_S$ in the pinned environment produced the same $y_S$, an observation about those $j$ runs, not a determinism proof;
- agreement with a named target: the authors released outputs or per-item scores $T$ for exactly $I_S$, pinned in $V$ by hash, and $\rho(y_S, T)$ holds.
None of the three is a reproduction of a reported result.
On uncertainty: Pineau et al. observe that "statistical analysis is seldom conducted in ML research".4 The NeurIPS checklist items include "Clearly defined error bars." and "A description of results with central tendency (e.g. mean) & variation (e.g. stddev)."4 Among NeurIPS 2019 submitters, 87% saw value in clearly defining metrics and statistics, yet 36% judged error bars not applicable to their results.4 If you report a mean over $s$ seeds, report the spread and $s$. A spread measured after the runs is a description, not an acceptance rule; $\rho$ must be fixed first. Fixed inputs and a fixed script do not imply a deterministic stage: the scorer may sample, call an external service or depend on unpinned packages. Claim determinism only if you verified it under the pinned environment, and state how. Otherwise report $j$ identical reruns as an observation of $j$ runs. In neither case present zero observed spread as an uncertainty estimate. Report the subset size $n$; a subset score is not an estimate of the full-benchmark score unless you justify that separately.
On code: "Having a copy of the code used to produce the experimental results is not a guarantee that this code is correct".4 Rerunning released code tests reproducibility in the narrow sense; it does not test robustness.
On baselines: Lipton and Steinhardt report that Melis et al. showed a series of published improvements "originally attributed to complex innovations in network architectures, were actually due to better hyper-parameter tuning", with vanilla LSTMs topping the leaderboard on equal footing.3 Document the tuning budget for your baseline and your method, or mark it unknown. Note also their caveat that ablation "is neither necessary nor sufficient for understanding a method, and can even be impractical given computational constraints."3
On outcomes: "There is rarely such a concise outcome to a reproducibility study", and the authors "have not found that the findings of most reproducibility studies lend themselves to such a coarse summary" as a single score.4 Report per result.
On the field's base rate: Pineau et al. summarise a separate replication study (Raff, 2019) and note that "a selection bias could be at play" in its comparison of attempts with and without author help.4 This module has not opened that study, so it does not repeat its figures; read them in the original. Pineau et al. conclude that it "remains to be established if the field is having a reproduction problem similar to the other fields, or if it would be better described as a reporting problem."4 Do not use any field-level rate as a prior for your specific paper.
Inaccessible reproduction and the substitute
Pineau et al. record two structural barriers: datasets that "cannot be released for legitimate privacy reasons", and experiments that "may require so much computation (time & number of machines) that it is impractical for any reviewer, or in fact most researchers, to attempt reproducing the work."4 For the first they describe a mitigation: "provide complementary empirical results on an open-source benchmark dataset, in addition to the results on the confidential data."4
The capstone's equivalent is the stranger-runnable substitute, a "constrain" answer at design time. If full reproduction of claim $C$ is inaccessible, define a narrower claim $C_s$ such that $C_s$ is supported by a check $S$ with:
- pinned inputs $I_S$ (for example a frozen accessible subset, or a released artifact such as model outputs, a checkpoint or a results file) and environment, all in $V$;
- a stated baseline with its own pinned inputs;
- a matched reference $r$ for $I_S$, or else $C_s$ restricted to completion, observed repeatability or agreement with a named released target $T$ for $I_S$ (see above);
- expected outputs $\hat{y}_S$ and the comparison rule $\rho$, both in $V$;
- instructions executable without the author.
Then someone other than the author must actually execute $S$ on $V$ and record $(y_S, D)$. The artifact may claim only $C_s$; $C_s \not\Rightarrow C$. Gate 1 then requires:
\text{Pass}_1(V) \iff \exists\, g \in \mathcal{R}_1(V) :\; g.\text{actor} \neq \text{author} \;\land\; \text{completed}(S, V)
\;\land\; \rho(y_S, \hat{y}_S) \;\land\; \neg\exists\, d \in g.D : \text{undermines}(d, C_s) \;\land\; g.\text{answer} = \text{allow}.
A run that fails on a missing dependency is not completed; the record answers deny, the author fixes the instructions, and a new $V'$ is rerun. If no admissible $S$ exists, record the limitation and leave the artifact private and draft. A limitation record is not a rerun record, so $\text{Pass}_1$ is false.
This is the misconception 500.P-a corrects: a documented access limitation, however honest, does not satisfy the independent-rerun gate.
Route 2: governance regime
For the pinned $V$, supply the dated role and use; legal-scope assumptions, labelled as assumptions; a clause → gate → evidence → owner table; a challenge and remedy walkthrough; and two distinct tests:
- Fail-closed test. Build a disposable, labelled synthetic variant $\tilde{E}$ from one row's evidence by a scripted perturbation $\pi$ that removes the evidence. Pin $\pi$, $\tilde{E}$ and the expected answer (hold or deny) in $V$. Never delete or modify the pinned original evidence. The test passes if the gate does not allow the action on $\tilde{E}$.
- Decision trace. From the intact packet, someone other than the author traces one complete decision clause → gate → evidence → owner and records whether every link resolved by hash.
The fail-closed test shows that missing evidence blocks action. The decision trace shows that a decision can be reconstructed. Neither implies the other.
Make no certification claim from a standard's catalogue scope.
Raji et al.'s SMACTR framework (Scoping, Mapping, Artifact Collection, Testing, Reflection) is a reference design for the evidence column: "Each stage of the audit yields a set of documents that together form an overall audit report".6 Its Reflection stage includes a remediation and risk mitigation plan and an algorithmic design history file, which "should assist with an audit trail, enabling the reconstruction of key decisions and events during the development of the product."6 Your decision trace, not the fail-closed test, is the check on that reconstruction property for one decision.
Raji et al. also flag an open problem your regime must face explicitly: thresholds. "If auditors discover, for example, unequal classifier performance across subgroups, how close to parity is necessary to say the classifier is acceptable?"6 State your threshold and its owner, or mark it unresolved.
Route 3: red-team report
For the pinned $V$, supply authorisation; reproducible tests on fictional transcripts (course-owned synthetic fixtures only); a stop and coverage log (tested, untested, stop conditions fired); an independently recorded ruling per finding; the mitigation and its retest; and residual limitations. The tester cannot rule on their own finding.
The independence requirement has a documented basis. Raji et al.: "Internal auditors necessarily share an organizational interest with the target of the audit." They warn that audits risk "becoming simply acts of reputation management for an organization", and position internal audits as "only one important aspect of a broader system of required quality checks and balances."6
Pre-submission checklist (in class)
| Item | Pass condition |
|---|---|
| Route | One of three, declared |
| $V$ | Manifest or immutable revision IDs covering $A \cup E$, including the environment |
claim, falsifier | Present, bounded, consistent with each other; stage re-executed named |
| $\rho$ | Comparison rule and matched reference pinned before any run (route 1) |
provenance | Every input with source and version |
stranger_check | Executable without the author; expected outputs stated |
| Route-specific items | All present, or explicitly marked missing |
limitations | Missing evidence, unchecked items, uncertainty; labelled and owned |
| Substitute (route 1) | $S$ defined, or "none: private and draft" |
| Privacy | No personal or protected data, no private-repository content, in any shareable file |
| Status | private, draft |
The privacy row matches the worksheet's offline verification step, which fails a bundle with "payload content inside (prompt, output, personal data)".2
The out-of-class gates
| $k$ | Gate | $\text{Allowed}_k$ | allow requires |
|---|---|---|---|
| 1 | Independent rerun or substitute rerun | $\neq$ author | $\text{Pass}_1(V)$ above |
| 2 | Red-team ruling | Red-teamer $\neq$ author; ruler is a named person $\neq$ tester | Every finding against $V$ has a ruling |
| 3 | Closure, retest, independent source re-check | Retester and source checker $\neq$ author | Every accepted finding closed with a passing retest on $V$; every affected claim re-checked as supported |
| 4 | Two-human review | One subject expert and one audience reader; not automated grading | Both dispositions favourable for $V$ |
| 5 | Privacy clearance | Named person | $V$ cleared; exclusions confirmed |
| 6 | Release decision | Bob and Ken | A decision to release naming $V$ |
Gate 1 operationalises the worksheet's independence premise.2 Its offline verification step asks a verifier to follow a bundle's VERIFY.md without the Foundation's own tool or an account, and ends in a one-sentence statement under the verifier's name.2 Its premise-gate example assigns approval to an independent validator rather than the team that wrote the rule.2 The NeurIPS 2019 reproducibility challenge stated the same goal at community scale: "to provide independent verification of the empirical claims in accepted NeurIPS papers, and to leave a public trace of the findings from this secondary analysis."4 It drew claims on 173 papers from participants at 73 institutions, and its reports went through a review cycle by NeurIPS reviewers to select a subset for publication.4
Tooling status is time-sensitive. As of November 2026, the worksheet's status line reads "Fall 2026 · live works today; fall lands in P2.", and it lists a one-command offline verify as the fall version of a step that is done manually today.2 A gate may not depend on tooling that is not yet live; this capstone requires no product or account.
Evidence that the process helps is itself limited. When NeurIPS reviewers were asked "Were the Reproducibility Checklist answers useful for evaluating the submission?", 34% responded Yes.4 Out of how many: this module did not check the number of reviewers who answered, so read the figure as a share of respondents, not of all reviewers. Pineau et al. state "we do not have concluding evidence that these processes indeed have an impact on the quality of the work or of the papers that are submitted and published."4 They add that reproducible results require "cultural and organizational changes, not just a code submission policy or a checklist."4 Treat the gates as necessary structure, not as proof of quality.
Material change and reopening
Define a material change $\Delta: V \to V'$ as any change to the claim, baseline, data, code or evidence, including changes made while closing red-team findings. For $V'$:
- Pin $V'$.
- Reopen affected gate-1 reruns and gate-2 rulings on $V'$.
- Close and retest accepted findings on $V'$; independently source-recheck affected claims.
- Obtain both reviewers' dispositions on $V'$.
- Obtain privacy clearance on $V'$.
- Bob and Ken record a release decision naming $V'$.
Impact records for unaffected checks
Any change to $A \cup E$ produces a new manifest, so old records cover only the old $V$. For $k \in \{1, 2, 3\}$, the gate's owner (never the author) makes an explicit, version-bound impact decision on $V'$. The records that can satisfy gate $k$ for $V'$ are
\mathcal{R}_k(V') = \{\text{fresh } g : g.k = k,\; g.V = V',\; g.\text{actor} \in \text{Allowed}_k\} \;\cup\; \{\, i : i.k = k,\; i.V = V',\; i.\text{actor} \in \text{Allowed}_k,\; i.\text{answer} = \text{allow},\; i.\text{basis} = \text{diff}(V, V'),\; i.\text{ref} \ni g_{\text{old}},\; g_{\text{old}} = \text{In}_k\big(V, i.\text{head}_V\big),\; \text{ok}_k(g_{\text{old}}, V),\; \neg\,\text{Denied}_k\big(V, i.\text{head}_V\big) \,\}.
Both arms keep the restrictions that define $\mathcal{R}k$: a record counts for gate $k$ only if it names gate $k$ and was made by an actor in $\text{Allowed}_k$. These filters apply before $\text{In}_k(V', N)$ picks the record with the highest sequence number, so a later record for another gate, such as the gate-6 release decision, is never a candidate for gates 1–3. An impact record $i$ states that $\text{diff}(V, V')$ does not touch what gate $k$ checked, and why. The record it cites must itself pass: $i.\text{head}_V$ is the position of $V$'s log that $i$ records as read when it is made, $g{\text{old}}$ is the record in force for gate $k$ on $V$ at that position, and it answers allow. If $V$'s log is not well formed up to $i.\text{head}_V$, no impact record can cite it. A deny or hold on $V$ cannot ground an impact allow on $V'$, even if $\text{diff}(V, V')$ leaves the inputs, metric and tolerance unchanged; neither can an allow on $V$ that a later record superseded, nor an allow recorded at gate $k$ on $V$ after a deny there. If gate $k$ on $V$ was never passed, gate $k$ on $V'$ needs a fresh record. Its rationale must cover two separate conditions:
- execution: $\text{diff}(V, V')$ changes nothing gate $k$ executed or read (instructions, inputs, code, environment, $\rho$);
- warrant: the old result still bears on the claim in $V'$. Write the claim as $(\text{population}, \text{metric}, r, \rho, \text{stage})$ plus prose. If $\Delta$ changes any element of that tuple, the old result tests a different claim and only a fresh record can satisfy gate $k$. Removing an unsupported inference while leaving the tuple unchanged can meet this condition; identical commands alone do not.
The old record $g_{\text{old}}$ is cited as evidence; it is not itself an approval of $V'$. If the owner judges the change affects their check, only a fresh record can satisfy the gate. For $k \in \{4, 5, 6\}$, $\mathcal{R}_k(V')$ contains only fresh records: review, privacy clearance and release are always redone on the exact release version.
This corrects the misconception 500.P-b targets: no rerun, ruling, review, clearance or release decision on $V$ authorises publication of $V'$. An impact record is a new decision about $V'$, not a carried-forward pass.
Raji et al. motivate checking during development rather than after the fact: external audits "tend to be conducted after model deployment, when the system has already negatively impacted users".6 Version binding is the same principle applied to review: the checks must cover the object that is actually released.
Worked trace
A fictional Big Widget researcher, Ana, targets one evaluation result in a placeholder paper (not a real paper; no real numbers are used). Training and full evaluation are compute-bound. The authors released the method M and baseline checkpoints, a frozen evaluation subset, the scoring script, and an appendix table that reports each model's score on that subset (the matched reference $r$).
| Step | Version | Event | Effect |
|---|---|---|---|
| In class | $V_1$ | $C_s$: "running the released M and baseline checkpoints on the released frozen subset, with the pinned scorer and environment, gives the appendix's subset scores for each within tolerance $\rho$; M scores higher there. Inference and scoring only; no claim about training or the full benchmark." $\rho$, $r$, falsifier, manifest, instructions, $n$ and limitations pinned | Checklist complete; private, draft |
| Gate 1 | $V_1$ | Ben (not the author) runs; fails on a missing dependency pin | Record answers deny on $V_1$. Instructions fixed, $V_2$ pinned; Ben reruns, both scores satisfy $\rho$; two repeats were bit-identical, recorded as an observation of two runs (determinism not claimed); answers allow on $V_2$ |
| Gate 2 | $V_2$ | Red-teamer finds the appendix gives the baseline a smaller tuning budget, so the claim invites crediting M's design for the gap; named ruler (not the tester) accepts | Accepted finding open on $V_2$ |
| Gate 3 | $V_3$ | Claim narrowed to add "tuning budgets differ, so this does not show what caused the gap": material change | $V_3$ pinned. Ben records an impact decision on $V_3$: not affected. Execution: $\text{diff}(V_2, V_3)$ touches only claim text. Warrant: the added sentence removes an unsupported causal inference; subset, metric, $r$, $\rho$ and stage are unchanged. Cites his gate-1 allow on $V_2$, still the record in force for $V_2$ (his $V_1$ deny could not ground it). Finding retested and closed on $V_3$; changed sentence source-rechecked on $V_3$ |
| Gates 4–6 | $V_3$ | Two favourable dispositions after closure; privacy clearance; Bob and Ken's decision | $\text{Release}(V_3, N)$ true for the log as it stands at publication, with no later hold or deny on $V_3$; $V_1$, $V_2$ never released |
Counterfactuals: had no released artifact existed, there would be no admissible $S$; Ana records the limitation, $\text{Pass}_1$ is false for every version, and the artifact stays private and draft. Had the paper reported only a full-benchmark score, there would be no matched $r$, and $C_s$ would shrink to completion and observed repeatability, or to agreement with a named released target $T$ for the subset if the authors had released one.
What a release does and does not establish
A release establishes that one bounded claim at one version survived independent rerun, red team, closure, two-human review and privacy clearance. It is evidence, not a certificate. It does not establish the full paper's claim, the lawfulness of a regime, or a system's safety. It is not a degree or credential: the Foundation states "these programs are not degrees".1 The worksheet keeps person and system apart: "Systems get an OVERT AAL and a coverage number from their own evidence."2
Three-yes check
- Get to Yes: before any gate, the five rubric fields, $\rho$ and route items, at a pinned $V$, with an admissible substitute (constrain) or an explicit private-and-draft deny.
- Stay at Yes: six gate records in force in a well-formed log at the moment of publication, each naming $V$, each answering allow, strictly ordered by sequence number, with actor constraints met, no later hold or deny superseding any of them, no deny on $V$ anywhere in the log, and no private content in any record.
- Recover to Yes: a hold names its owner and can be released on the same $V$, after which every later gate decides again. A deny is final for $V$; it, or any $\Delta: V \to V'$ (including a fix that closes an accepted finding), sends the work to $V'$. Pin $V'$, take an impact decision where eligible or a fresh record for gates 1–3, redo gates 4–6, and only then is $\text{Release}(V', N)$ possible. An announced fix is not a return to yes.