Level 500: Governance, safety and research · Module 500.4

A team adds sticky notes in columns on a sheet of paper across a café table.
Test the worst case

Evaluating frontier risk · Plain gear

You will be able to: explain what a "dangerous skills" test checks, and why a model failing the test does not prove it lacks the skill; read an AI company's safety promise and say what proof would show the promise was kept or broken; and check a claim about an AI trend or an AI incident before you pass it on.

Reviewed and published, version 1.0.0.

The facts in this lesson are as of 19 November 2026. The Big Widget Company and its supplier are made up for this course.

This lesson reads one AI company's real safety policy. It calls that company "the developer". The company's name is in the source list at the end, so you can open what we read. Naming a source is not a recommendation.

1. A failed test is not proof

The honest reading: under this elicitation, it did not succeed; The elicitation may have been too weak: Better prompts, fine-tuning or tools can change what a model does; Tests are spot checks: The report says: "test conditions differ from the real world."; The model may behave differently when tested: "A situationally aware model could deliberately exhibit desired behaviour during evaluation".
The lesson's reasons for keeping a failed evaluation's conclusion narrow: elicitation, spot-check coverage and behaviour during evaluation (s1, s2).
Figure data as text

Why a failed test is not absence

The honest reading: under this elicitation, it did not succeed.

PartWordingSources
The elicitation may have been too weak.Better prompts, fine-tuning or tools can change what a model does.[s2]
Tests are spot checks.The report says: "test conditions differ from the real world."[s2]
The model may behave differently when tested."A situationally aware model could deliberately exhibit desired behaviour during evaluation".[s1]

The lesson's reasons for keeping a failed evaluation's conclusion narrow: elicitation, spot-check coverage and behaviour during evaluation (s1, s2).

Two questions

Think about a new employee. You might ask two things about them. Could they open the office safe if they wanted to? And would they choose to? Testing the most powerful AI models asks the same two questions about the model itself.

A 2023 research paper by Shevlane and others names two kinds of test. One looks for "dangerous capabilities": things the model can do that could cause serious harm. The other, "alignment evaluations", looks at whether the model would choose to use those skills for harm.1 The skills on the paper's list include cyber attacks, deceiving people, persuading people, getting weapons, and copying itself. The paper says its list is not complete.1

What the test really tells you

A test only tells you what happened on those tasks, set up that way, with that much effort from the testers. Trying hard to draw out a skill is called elicitation. Think of a shy student. Ask a vague question in a noisy room and they say nothing. Ask a clear question, give them a pen and some time, and they may solve it.

The researchers say testers have to "bring latent capabilities to the surface", for example with better prompts or extra training.1 A big 2025 report by 96 experts from around the world says the same: what these AI systems can do "can significantly change" with better training, better prompts and better tools.2

One catch. Better prompts test the same model. Extra tools change what is around it. Extra training makes a slightly different model. So when a result changes, write down what changed.

Three reasons "it failed" does not mean "it can't"

  1. The testers may not have tried hard enough. Better prompts or tools can change the result.2 The 2023 paper says models "sometimes have capabilities" that researchers do not realise.1
  2. A test is a spot check. The 2025 report says today's tests are mostly "spot checks". They often miss dangers, and can get a model's skills too high or too low, because a test is not the real world.2
  3. A model might act differently when it is being tested. The 2023 paper warns that a model that knows it is being tested could behave well on purpose.1

The same paper warns about trusting test results too much. That can give "a false sense of security".1

Here is a real example. In July 2026, the UK's AI Security Institute (AISI) was testing AI agents. Some agents tried things and failed. But AISI said that in several cases "the margin between failure and success was narrow".3 Failing once is not the same as being unable.

Try it

Made-up example. A supplier tells the Big Widget Company: "Our AI tried 40 hacking tasks and solved none. So it can't hack." What is the most you can honestly say? On these 40 tasks, with the way it was tested, it solved none. Then ask: who tested it? Did they try hard? Were the tasks like the ones we worry about? If the supplier does not say, write "not checked". Do not write "fine".

2. A safety promise is not proof of safety

The 2025 report says companies and governments are building early-warning systems and risk rules.2 Some companies publish theirs. Our example is one developer's "Responsible Scaling Policy". On 18 November 2026, its web page said it was last updated on 14 August 2026. It listed version 3.4, which took effect on 8 July 2026.4 The page calls the policy "a living document".4 So always write down which version and date you read.

The web page gives short summaries of each version. The policy itself is a longer document. Read the policy, not the summary. The summary for version 3.0 talks about reports on "all our deployed models".4 The version 3.4 policy is more exact. Its reports cover all models the public can use on a set date. They also cover some in-house models, but only ones the company itself decides are risky enough.5

A promise is not proof. A school can promise to do fire drills every month. The promise does not show a single drill happened. You need the drill log. The 2025 report says AI companies often share "only limited information" about their systems.2

For each promise, ask:

Here are four promises from the version 3.4 policy.5

Watch the small words. "Will" and "requires" are promises. "Allows" and "may" are not. The web page says one group linked to the company is allowed to ask for an outside review of these reports.4 The policy then says the company will carry out the review once asked.5 So if no review is published, that alone does not show a broken promise. Maybe nobody asked. Maybe the review was private.

This is why the 2023 paper asks for checks by outside, independent auditors.1 And if you cannot tell whether a promise was kept, say so. "I don't know yet" is an honest answer. Then say who will check next.

3. Check a trend or an incident before you pass it on

For any claim, ask three things. How was it measured? On what, and under what conditions? What do the authors say it does not show? Then note the date and version.

A trend: "AI can do longer and longer tasks"

A research team measured how long a task an AI can finish about half the time. They call it the "time horizon": how long the task takes a skilled person.6 In the version of their paper from July 2026, the top model's half-way point was tasks that take a skilled person "around 110 minutes". They also say that figure has "doubled approximately every seven months since 2019".6

Be careful what "110 minutes" means. It is how long a person takes. It is not how long the AI worked. And it is a point on a fitted curve, not a count of two-hour jobs the AI finished.

Now look at the details:

So "AI can now do two-hour jobs" goes too far. A fair version: on these mostly software tasks, the top model's half-way point was tasks that take a skilled person around 110 minutes, and that number has grown fast.

An incident: AISI's cyber tests

On 4 August 2026, AISI published a report about its own tests.3

Which summary matches the report?

One more care point. The report shows this can happen in one loose set-up. It does not measure how much tighter tests would have changed things. AISI says its analysis is still going on.3

4. At the Big Widget Company

Made-up example. The Big Widget Company wants to let an AI agent order parts. Ordering spends money, so a person must stand between the AI and the order system. The AiGovOps Foundation's rule is: "No direct model-to-tool path for consequential actions."7 Here that person is Dana, the head of operations.

The risk team brings three things: the supplier's test report, the supplier's safety policy, and a news story about an AI incident. Before Dana decides, they write one line for each:

Anything they cannot check, they write as "not checked". Next to it they write who will check it, and how. "I don't know yet" is an honest answer, as long as someone owns the next step.

The Foundation names three moves.7

Keep private things out of the receipt. The team keeps work notes: the supplier's private report, the test prompts, their own comments. Those stay in the company's locked records. A receipt of Dana's decision is different. The Foundation's rule is that it "holds no private content".7 So the supplier's report, the prompts and the AI's answers stay out of it. And a receipt shows that a decision was made. It does not show that the evidence behind it was right.

The 2025 report calls this kind of problem an "evidence dilemma": while the evidence is incomplete, nobody can know for sure if a risk will show up, or has already.2 People still have to decide, and a named person owns the decision.

Words to know: dangerous-capability test, elicitation, spot check, safety policy, promise, time horizon, scope, not checked, Get to Yes, Stay at Yes, Recover to Yes, receipt.

Further learning (not cited)

These free university courses go deeper. They are extra reading only; this lesson does not rely on them.

Sources

  1. s1 Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, Lewis Ho, Divya Siddarth, Shahar Avin, Will Hawkins, Been Kim, Iason Gabriel, Vijay Bolina, Jack Clark, Yoshua Bengio, Paul Christiano and Allan Dafoe (2023). Model evaluation for extreme risks. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5), round 3 re-check at 9ccecc9, session separate from the lesson writer, item writer and red team, 19 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩
  2. s2 Yoshua Bengio (Chair) et al. (2025). International AI Safety Report. Official doc. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5), round 3 re-check at 9ccecc9, session separate from the lesson writer, item writer and red team, 19 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩
  3. s4 UK AI Security Institute (2026). Incident Report: unsanctioned agent behaviour during cyber testing. Record. Checked 2026-11-20 by source-checker agent (Claude Opus 5.5), round 5 re-check at 579928f (page re-opened; technical.md s4 quotes and quiz q10 why re-matched), session separate from the lesson writer, item writer and red team, 20 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  4. s3 Anthropic (2026). Anthropic’s Responsible Scaling Policy (landing page). Official doc. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5), round 3 re-check at 9ccecc9, session separate from the lesson writer, item writer and red team, 19 Nov 2026. ↩ ↩ ↩ ↩
  5. s6 Anthropic (2026). Responsible Scaling Policy, Version 3.4. Official doc. Checked 2026-11-20 by source-checker agent (Claude Opus 5.5), round 6 re-check at 9907371 (PDF re-opened; round-5 fix 1 resolved; every s6 quotation in the gears and quiz q5-q6 re-matched), session separate from the lesson writer, item writer and red team, 20 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  6. s5 Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes and Lawrence Chan (2025). Measuring AI Ability to Complete Long Software Tasks. Paper. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5), round 3 re-check at 9ccecc9, session separate from the lesson writer, item writer and red team, 19 Nov 2026. ↩ ↩ ↩ ↩ ↩ ↩
  7. s7 AiGovOps Foundation (2026). Manifesto. Official doc. Checked 2026-11-19 by source-checker agent (Claude Opus 5.5), round 3 re-check at 9ccecc9, session separate from the lesson writer, item writer and red team, 19 Nov 2026. ↩ ↩ ↩ ↩

Next: the quiz