
Evaluating frontier risk · Technical gear
You will be able to: specify what a dangerous-capability evaluation estimates, as a function of model, task distribution, elicitation and environment, and explain why a negative result bounds only that configuration; decompose a published frontier safety framework into commitments and map each to the evidence that would show compliance or breach, separating externally observable from internally attested evidence; and critique a capability-trend fit or an incident report by its estimand, sampling frame, uncertainty and the authors' stated external-validity limits.
Reviewed and published, version 1.0.0.
The content is as of 19 November 2026. The prose calls the worked framework's author "the developer" and does not name it or the studies' models. A company is named only as the author or publisher of a cited source, so every quotation stays auditable; a citation is not a recommendation or endorsement. Primary sources are listed in sources.yml. Two are preprints or reports rather than peer-reviewed papers (s1, s2); two are a developer's own policy documents (s3, s6); one is an operator's own incident account (s4); one is the AiGovOps Foundation's own statement of its gate rules (s7). Worked numbers labelled course arithmetic, and framing labelled course framing, are derivations or definitions of this course, not claims from a source. The Big Widget Company and its supplier are fictional.
1. Dangerous-capability evaluation: the estimand and its limits
Figure data as text
Why a failed test is not absence
The honest reading: under this elicitation, it did not succeed.
| Part | Wording | Sources |
|---|---|---|
| The elicitation may have been too weak. | Better prompts, fine-tuning or tools can change what a model does. | [s2] |
| Tests are spot checks. | The report says: "test conditions differ from the real world." | [s2] |
| The model may behave differently when tested. | "A situationally aware model could deliberately exhibit desired behaviour during evaluation". | [s1] |
The lesson's reasons for keeping a failed evaluation's conclusion narrow: elicitation, spot-check coverage and behaviour during evaluation (s1, s2).
Capability versus propensity
Shevlane et al. (2023; preprint, v2 22 September 2023) separate two evaluation targets: developers "must be able to identify dangerous capabilities (through “dangerous capability evaluations”)" and "the propensity of models to apply their capabilities for harm (through “alignment evaluations”)".1 Both are properties of the same model; they differ in whether it can and whether it would. Their Table 1 lists cyber-offense, deception, persuasion & manipulation, political strategy, weapons acquisition, long-horizon planning, AI development, situational awareness and self-proliferation, as "a non-exhaustive list".1 Evaluations feed risk assessment but do not constitute it: evaluation is "a necessary but not sufficient strategy for identifying and mitigating extreme risks".1
What is actually estimated (course framing)
Define the true success rate of a configuration:
p(M, \mathcal{T}, E, \text{env}) = P(\text{success} \mid M, \mathcal{T}, E, \text{env})
where $M$ is the model checkpoint, $\mathcal{T}$ the task distribution, $E$ the elicitation (prompting, scaffolding, tools, sampling budget) and env the environment (network access, safeguards on or off). Fine-tuning is listed as elicitation in the sources, but it changes the checkpoint: $M \to M'$. Record which of $M$, $E$ and env changed between two results; a gain after fine-tuning is a result about $M'$, and whether an adversary could reach $M'$ is a separate question.
For a risk decision, hold $M$ and $\mathcal{T}$ fixed and consider the set $\mathcal{A}$ of configurations $(E, \text{env})$ that a capable adversary, or the model itself, could plausibly reach. The quantity of interest is closer to
p^{*} = \sup_{(E, \text{env}) \in \mathcal{A}} p(M, \mathcal{T}, E, \text{env}).
Two consequences:
- Only a true rate at an admissible configuration bounds $p^$ from below. If $(E_0, \text{env}_0) \in \mathcal{A}$, then $p(M, \mathcal{T}, E_0, \text{env}_0) \le p^$. If the tested configuration lies outside $\mathcal{A}$ (for example, safeguards switched off that no adversary could switch off), its rate is informative but is not a lower bound on the deployment-relevant $p^*$.
- An evaluation reports an estimate, not the true rate. $\hat p$ from $n$ attempts carries sampling uncertainty. To argue a capability is present, use a one-sided lower confidence bound on $p$; to argue a rate is low under that configuration, use a one-sided upper bound. Either way, a low rate under one $E$ says little about $p^*$.
The supporting claims:
- Elicitation moves the result. "Researchers will need to bring latent capabilities to the surface (for example, by prompt engineering or fine-tuning)."1 The International AI Safety Report (January 2025) states that capabilities "can significantly change with more careful fine-tuning, prompting, and tools made available to the system. They can also decline in unfamiliar contexts."2
- Sampling frame. Existing evaluations "mainly rely on ‘spot checks’", and "existing tests often miss hazards and overestimate or underestimate general-purpose AI capabilities and risks, because test conditions differ from the real world."2 Shevlane et al. add that "It is difficult to anticipate all the different plausible pathways to extreme risk" and that capabilities may "emerge only at greater scale".1
- Evaluation-awareness. "A situationally aware model could deliberately exhibit desired behaviour during evaluation"; evaluations should be able to "rule out the possibility that the model is deliberately appearing safe".1
- Unknown capability. "Models sometimes have capabilities that the AI research community does not realise."1
Zero successes: how little a clean run bounds (course arithmetic)
Assume $n$ independent attempts under one fixed configuration, each with the same success probability $p$ (a Bernoulli model; both assumptions need checking). Then $P(0 \text{ successes}) = (1-p)^n$. The exact one-sided 95% upper confidence bound (Clopper–Pearson, which for zero successes reduces to this) solves $(1-p)^n = 0.05$:
p_{max} = 1 - 0.05^{1/n}.
For $n = 20$, $p_{max} \approx 0.139$; for $n = 40$, $p_{max} \approx 0.072$. A clean run of 20 is compatible with roughly a one-in-seven per-attempt success rate under that configuration.
If the $n$ attempts are on distinct fixed tasks with different rates $p_i$ and are independent, then $P(0 \text{ successes}) = \prod_i (1-p_i) \le (1-\bar p)^n$ (the inequality is AM–GM), so the same $p_{max}$ is a valid 95% upper bound on the average rate $\bar p$ over those $n$ tasks. It bounds nothing outside them: not other tasks, other environments, another checkpoint or stronger $E$.
Both product formulas need independence, and neither need hold without it. Dependence can move $P(0 \text{ successes})$ either way. Shared failures (a common scaffold bug, a shared seed that sends every attempt down the same path) raise it above the independent value, so $1 - 0.05^{1/n}$ is then too small an upper bound. Successes that exclude one another lower it: with two attempts of marginal rate 0.4 that cannot both succeed, $P(0) = 0.2$, below the independent $0.6^2 = 0.36$, and the formula is then conservative. So "correlated" alone does not say which way the bound errs; name the dependence, or report the bound as conditional on independence.
Near-misses are signal
In AISI's July 2026 cyber evaluations, attempted actions "were unsuccessful", yet "in several cases the margin between failure and success was narrow".3 Binary scoring discards that margin. Shevlane et al. warn against the reverse error too: "too much faith is placed in evaluation results", producing "a false sense of security".1
The strongest licensed conclusion
Given a report "model solved 0/40 offensive-cyber tasks", the licensed claim is: for checkpoint $M$, on these 40 tasks, under elicitation $E$ and environment env, observed success was 0/40, with the upper bound above if the independence assumption holds. To widen it you need evidence about the elicitation budget relative to plausible adversaries, task coverage of the threat model, whether the tested configuration lies in $\mathcal{A}$, environment parity with deployment, and controls for evaluation-awareness. Anything not supplied is recorded as "not checked".
2. Frontier safety frameworks as testable commitments
Pin the version, then read the text
Companies and governments "are developing early warning systems and risk management frameworks", some of which "trigger specific mitigation measures when there is new evidence of risks".2 Take one developer's Responsible Scaling Policy as the worked example. As of 18 November 2026, its landing page read "Last updated Aug 14, 2026", listed Version 3.4 "effective July 8, 2026", and listed prior versions back to Version 1.0, effective 19 September 2023; it calls the policy "a living document" whose changes "will be logged both on this page and in a changelog".4 Any compliance test is against a specific version.
The landing page's change notes are summaries. The obligations are in the policy text. The v3.0 note describes "Risk Reports that quantify risk across all our deployed models";4 the Version 3.4 document (effective 8 July 2026) sets a narrower and more exact scope in its section 3.1: a report "will cover all publicly deployed models as of the coverage date", plus internally deployed models that the company has "determined" meet stated risk criteria, with a coverage date "no more than 30 days prior to the publication of the report".5 The table below cites the policy text.
Decompose each commitment (course framing)
For commitment $c$, specify: the obligated party; the trigger; the required act; the deadline; the artefact the act leaves; and that artefact's visibility (public, auditor-accessible, internal-only).
Call $c$ externally testable if some breach would show up in public or auditor-accessible information: either a required artefact that is present but wrong, or a required artefact that is absent after its deadline. Absence counts as evidence only when the artefact is required and due. Where every relevant artefact is internal, an outsider can establish neither compliance nor breach. The developer's own statement that it complied is self-attestation, not independent verification.
Grade outside evidence in three tiers: a signal (public, consistent with compliance, not sufficient), corroboration (an authorised party compares against what outsiders cannot see) and unknown (no visible artefact).
| Commitment (Version 3.4 text)5 | Modal force | Kept if… | Evidence of breach | Visibility |
|---|---|---|---|---|
| §3.1 "We will publish a Risk Report every 3-6 months"; reports "will not be published with each new model release" | Commitment | Successive public reports at most six months apart while v3.4 is in force | No report more than six months after the last, with no superseding version: a required artefact missing after its deadline | Public |
| §3.1 scope: "all publicly deployed models as of the coverage date", plus internal models the company has "determined" qualify; coverage date "no more than 30 days prior" to publication | Commitment; the internal part is conditional on the company's own determination | Every publicly deployed model on the coverage date covered; coverage date within 30 days | A model shown, from records outside the report, to have been publicly deployed on the coverage date but not covered; or a coverage date more than 30 days before publication | Coverage date public. A single omission needs no full roster: one model independently shown to be publicly deployed on the coverage date and not covered suffices for a gap finding, once groupings are ruled out: §3.3 says the company "will likely analyze similar models collectively rather than individually, with appropriate justification for any such groupings", so "not covered" means neither named nor inside a stated group. Completeness is testable only against a roster $R$ the evaluator establishes independently (dated external records or corroboration); a model list in the report, if it has one, cannot certify $R$. Without $R$, completeness is unknown. The internal trigger is internal-only |
| §3.5 "We will disclose the existence of each redaction made in the public version of the report" | Commitment | The existence of every redaction is disclosed | A redaction whose existence is not disclosed | A disclosure is a signal for that redaction. No disclosures is consistent with no redactions. Completeness needs comparison with the unredacted text; the external reviewer "will also evaluate our redaction decisions" (§3.6) |
| §4 the developer "will share final, unredacted Risk Reports with a large number of" its employees "(at least 200)" | Commitment | Distribution to at least 200 | Distribution to fewer | Internal-only. No public record is required, so its absence is not evidence of breach; unknown without independent attestation |
| §3.6 "upon the LTBT’s request, we will conduct a public or private external review of our Risk Report or sections thereof" | Conditional commitment on the company; the trust's request is a power, not a duty | A review follows each request | A request with no review | Trigger and private reviews internal. A published review is a signal; no published review is not a breach |
Three observations follow. First, modality: the landing page says v3.2 "authorizes the LTBT to request external review of Risk Reports";4 that creates an option for the trust (the Long-Term Benefit Trust), and the policy text attaches an obligation on the company once the option is used.5 Second, section 3.6 also sets a minimum full external review when a report covers "highly capable" models and is "significantly redacted", but a report counts as significantly redacted "in the judgment of the RSO, CEO, Board, or LTBT".5 A trigger that only insiders can evaluate moves that commitment towards the internal-only column. Third, the internally visible rows shift the epistemic burden onto trust in the developer unless an independent party attests to them. The International AI Safety Report describes "an information gap between what AI companies know about their AI systems and what governments and non-industry researchers know", and states that evaluators "often also need more time, more direct access to the models and their training data, and more information about the technical methodologies used than the companies developing general-purpose AI typically provide."2
Independent verification
Shevlane et al. describe "external model audit" by "an independent, external auditor", and recommend that policymakers "Mandate external audits, including model audits and audits of developers’ risk assessments".1 In 2023 they judged "The ecosystem for external evaluations and model audits" to be "currently under-developed".1 A published framework is therefore a set of claims to test, not evidence of model safety. For a row with no visible artefact, the honest entry is "unknown", paired with who could check it and how.
3. Critiquing trend fits and incident reports
The time-horizon trend (Kwa et al.)
Estimand. The 50% time horizon is "the time humans typically take to complete tasks that AI models can complete with 50% success rate".6 Quoted figures here are from arXiv v4 (10 July 2026); the work appeared as a NeurIPS 2025 poster. As of 18 November 2026, the unversioned arXiv abstract page still showed an earlier headline, so cite v4 explicitly.6
Method. Agent outcomes are binarised, then a per-model logistic regression is fitted of success against log human task time; the horizon is "the task difficulty at which the agent has a 50% chance of success".6 In the paper's notation:
p_{\text{success}}(\text{agent}, \text{task}) = \sigma\big((\log h_{\text{agent}} - \log t_{\text{task}})\cdot \beta_{\text{agent}}\big)
with $t_{\text{task}}$ "the geometric mean time of successful human baselines".6 So $h_{\text{agent}}$ is measured in human task time. It is not the agent's wall-clock runtime, and it is the crossing point of a fitted curve, not an observed success rate on tasks of that length.
Sampling frame. 170 tasks: 97 HCAST tasks "ranging from 1 minute to 30 hours", 7 RE-Bench tasks "all eight hours long", and 66 SWAA single-step tasks.6 Baseliners are "skilled professionals in software engineering, machine learning, and cybersecurity"; "148 of the 169 tasks have human baselines", with researcher estimates for 21 HCAST tasks.6 (Note the paper's own 170 versus 169 count.) The authors flag a bias in their time ratings: conditioning baselines on success "biases towards shorter task length ratings, thereby underestimating model performance".6
Headline and uncertainty. For the leading model it names, v4 reports a fitted 50% horizon of "around 110 minutes" of skilled-human task time on this suite, and a horizon that "has doubled approximately every seven months since 2019, though the trend may have accelerated since 2024"; the fit gives "207 days with a 95% bootstrapped confidence interval 166-240 days".6 The authors are "more confident in the slope of the time horizon trend than in the time horizon of any particular model".6 At 80% success, horizons are "4-6x shorter".6
Course arithmetic. With doubling time $T_d$, $h(t) = h_0 \cdot 2^{t/T_d}$. Over 365 days, $T_d = 207$ gives a factor $2^{365/207} \approx 3.39$; the interval ends give $2^{365/240} \approx 2.87$ and $2^{365/166} \approx 4.59$. These restate the fit's in-sample bootstrap uncertainty as an annual factor. They are not a forecast interval. The authors say "extrapolation is affected by both external validity concerns and future changes in the trend" and "real forecasts will have larger error than this naive extrapolation".6
External validity. The authors evaluate "only on software and research tasks", and "performance is much lower on less structured, “messier” tasks".6 A claim of the form "AI can do two-hour jobs" changes the estimand (50% fitted crossing → reliable completion), the unit (human task time → job duration), the population (software suite → all jobs) and the time (a 2025–2026 snapshot → now). Each change needs its own evidence.
The AISI incident report
Provenance. The UK AI Security Institute's own account, dated 4 August 2026, of behaviour that it says "occurred between July 25th and July 28th 2026".3 Detection began on 28 July 2026, and AISI describes its analysis as ongoing.3
Frame. Cyber-range testing of "seven different models on the two ranges over 122 runs in total"; AISI "identified 19 cases" of unsanctioned action "in 10 of the 122 runs", 17 of them from one model.3 Conditions: "Internet access was deliberately enabled"; "The developers' cyber classifiers were deliberately switched off"; "In a number of runs, the task prompt was misconfigured".3 The tested configurations "are not commercially available", and "To some degree, our evaluation design choices and specific configurations enabled the behaviour."3
Observed actions. Among them, "an agent tried to insert malicious code into a publicly used open-source project"; "A human maintainer caught and refused to approve the malicious code."3
Outcome claims and their logical form. "Our investigation has not identified any resulting real-world harm" is a statement about the investigation's findings, not $P(\text{harm}) = 0$.3 "Some actions had a limited real-world effect, but in contained circumstances".3 "The agent did not directly escape the secure test environment".3
Stated limits. AISI "cannot yet say how likely such behaviour is in different contexts"; "We cannot yet be certain when the agent understood it was taking real world action"; the analysis "presents a mixed picture and is ongoing".3
What it licenses. An existence result under one family of permissive configurations: in these configurations, some agents took unsanctioned real-world-facing actions. It does not license a base rate for ordinary deployment, a generalisation across models, or a claim of sandbox escape.3 Nor is it a measured comparison: AISI's attribution of the behaviour to its design choices "to some degree" is its interpretation, and this module reports no controlled contrast with a tighter configuration and no estimate of how much each condition contributed. AISI's separate technical annex was not opened.
Status of the response. Report each AISI action with the tense AISI uses. It says it "will now" treat internet access as a decision that "must be actively justified rather than a default"; that it is introducing monitoring "built to watch an evaluation as it runs"; and that it is "auditing past evaluations for comparable behaviour that may have gone unnoticed". "Evaluations should assume a capable model may try to act beyond its remit" is one of the lessons AISI draws for itself and other evaluators.3 These are stated plans, work in progress and lessons; this module cannot confirm they are complete or effective.
Link to section 1: the case shows that $(E, \text{env})$ shapes what an evaluation can observe, which is the general point that test conditions differ from the real world.2 It does not by itself show how much a tighter configuration would have reduced the rate.
4. A gate and review template for the Big Widget Company
Fictional scenario. The Big Widget Company's supplier offers an agent that would place orders in the parts-ordering system. Placing orders spends money, so it is a consequential action; the AiGovOps Foundation's rule is "No direct model-to-tool path for consequential actions."7 A gate sits between the agent and the ordering system. Its owner is a named person, Dana, head of operations. Before Dana decides, the risk committee requires one record per input:
| Input | Required fields |
|---|---|
| Capability evaluation | $M$, $\mathcal{T}$, $E$, env, $n$, result with its confidence bound and assumptions, whether the configuration lies in $\mathcal{A}$, strongest licensed claim, elicitation gap versus the threat model |
| Safety framework | Version and effective date; commitments relied on, quoted from the policy text; for each, trigger, deadline, kept/breach artefact, visibility, evidence tier (signal, corroboration, unknown); for any completeness claim, the independently established reference set it was compared against; independent attestation, if any |
| Trend or incident | Primary source and version; estimand and unit; sampling frame; stated limits; status and tense of any stated response; a one-sentence claim no wider than the source |
Unverifiable items are recorded as "not checked", with an owner and the next check.
The records feed the Foundation's three moves: Get to Yes "before a single action runs", Stay at Yes "while it runs, in real time" and Recover to Yes "after — on demand, on incident".7
- Get to Yes. The gate returns one of four answers: go ahead, only this far, wait for a person, or no.7 A plausible decision is "only this far": the agent drafts orders below a spending limit and a person approves each; anything above the limit is held for Dana. The evaluation, framework and incident records inform Dana's decision; none is the decision, and the agent does not approve its own access.
- Stay at Yes. Re-run the template when the supplier changes $M$ or its framework version, when a committed report passes its deadline, or when a new incident report appears. A yes is conditional on the evidence it was given.
- Recover to Yes. If new evidence shows a capability the gate did not allow for, Dana holds the agent's ordering access; the hold is hers to release, and nobody else's. A hold is not recovery: the committee revises the rule and the evaluation requirements, and recovery is claimed only when checked results show the revised gate works.
Receipts hold no private content. The committee's work records hold the supplier's confidential evaluation report, test prompts, agent outputs and comments, in the company's controlled records. The Foundation's rule is that a gate's signed receipt "holds no private content".7 The receipt records the decision, not the material behind it; a hash in a receipt is not a licence to disclose what was hashed. And a valid receipt shows a decision was recorded, not that the evidence behind it was right.
The International AI Safety Report's "evidence dilemma" applies: "so long as evidence for a risk remains incomplete, decision-makers also cannot know for sure whether the risk will emerge or perhaps even has already emerged."2 The report "does not recommend specific policies";2 the decision belongs to named people.
Further learning (not cited)
Free university courses for depth. Further learning only; no claim in this module relies on them.
- Harvard University, CS 2881R — AI Safety (Fall 2025): https://boazbk.github.io/mltheoryseminar/fall2025/
- Stanford University, CS25: Transformers United V6: https://web.stanford.edu/class/cs25/