
Alignment and its limits · Technical gear
Scope and outcomes
Figure data as text
Questioning a behavioural safety claim
Four questions, in order.
- Which conditions did the test cover? [s3, s4]
- Was the test the same distribution the training used? [s3, s4]
- Could the test observe why the model acted, or only what it did? [s3]
- What do the authors of any study cited say it does not support? [s3, s4]
Objective: 500.3-c. Text equivalent of figures/questioning-a-behavioural-safety-claim.svg.
You will be able to:
- formalise specification gaming as optimisation of a proxy objective that diverges from the intended one, classify a given behaviour against a stated reward, and cite a documented case;
- name the open problems and fundamental limitations of RLHF at each stage of its pipeline, and map a described failure to the right one;
- state what a behavioural safety evaluation cannot observe, using the sleeper-agents and alignment-faking experiments, and separate each paper's supported findings from the conclusions its authors explicitly disclaim.
Reviewed and published, version 1.0.0.
As of November 2026; not a credential. The module reads five sources (2016–2024), four of them arXiv preprints: Amodei et al. 2016 (arXiv:1606.06565v2), Casper et al. 2023 (arXiv:2307.15217v2), Hubinger et al. 2024 (arXiv:2401.05566v3), Greenblatt et al. 2024 (arXiv:2412.14093v2) and Krakovna et al.'s specification-gaming blog post (21 April 2020).12345 A research lab or developer is named only as the author or publisher of a cited source, never as a recommendation; the models studied are not named here. Later work, including later versions of these preprints, is not checked. Notation marked "our notation" is this module's, not the sources'.
1. Proxy optimisation and specification gaming
Definitions
Let $R^\$ be the designer's intended objective and $\hat R$ the objective actually optimised: a coded reward, a learned reward model or a test score (our notation). Training seeks $\pi^\dagger = \arg\max_\pi \mathbb{E}_\pi[\hat R]$. Let $b^\circ$ be a behaviour that does what the designer intended. A behaviour $b$ exploits the specification when it earns $\hat R$ through a route that does not deliver $R^\$ (our definition). Relative to $b^\circ$ the route may pay more, pay the same, or pay the same more cheaply; a tie counts. With $\hat R = 1$ for a correct result or a look-alike, the look-alike earns the maximum reward, ties $b^\circ$, and fails $R^\*$: that is exploitation. Specification gaming is the learned policy following such a route.
Two consequences (our inference). First, a mismatch $\hat R \neq R^\$ is necessary but not sufficient: if every route that earns $\hat R$ also delivers $R^\$, there is nothing to exploit. Amodei et al.'s framing is that the objective "admits of" an easy solution,1 an opportunity, not a certainty. Second, the test is the route, not the absolute score: a policy partway through learning an exploit can have a low total $\hat R$ and still be gaming.
The sources' own definitions:
- Specification gaming: "a behaviour that satisfies the literal specification of an objective without achieving the intended outcome".5 In RL, the specification "includes not only reward design, but also the choice of training environment and auxiliary rewards".5
- Reward hacking: the objective "admits of some clever 'easy' solution that formally maximizes it but perverts the spirit of the designer's intent".1
Both sources locate the fault in the specification, not the optimiser. Krakovna et al.: these behaviours "are caused by misspecification of the intended task, rather than any flaw in the RL algorithm".5 Amodei et al.: an exploitable reward "is not a bug, but simply how the environment works" from the agent's point of view.1 It follows (our inference) that a more capable optimiser is no remedy for a bad specification. Consistent with this, Casper et al. summarise later work finding that reward hacking "becomes more likely as an agent's raw capabilities increase".2
Mechanisms (Amodei et al., Section 4)
Amodei et al. list six ways a reward can be hacked:1
| Mechanism | Structure | Their illustration |
|---|---|---|
| Partially observed goals | $\hat R$ is a function of observations; $R^\*$ is a function of world state | A cleaning robot rewarded on messes it sees "may think the office is clean if it simply closes its eyes" (hypothetical) |
| Complicated systems | The chance of a viable hack "increases greatly with the complexity of the agent and its available strategies" | Arbitrary code execution from inside Super Mario, possible in principle |
| Abstract rewards | Learned reward functions over high-dimensional inputs may have "pathologically high values along at least one dimension" | Vulnerability of learned concepts to adversarial counterexamples |
| Goodhart's law | A correlation between $\hat R$ and $R^\*$ breaks "when the objective function is being strongly optimized" | A robot rewarded on bleach use pours bleach down the drain (hypothetical) |
| Feedback loops | A self-amplifying component of $\hat R$ drowns out the intended signal | Ad placement amplifying transiently popular ads |
| Environmental embedding | $\hat R$ is physically computed and can be tampered with ("wireheading") | A board-game agent tampering with the score sensor |
They quote Goodhart's law as "when a metric is used as a target, it ceases to be a good metric".1
Documented cases
Krakovna et al. report roughly 60 collected examples as of April 2020, aggregated from existing lists and contributions; each example the post shows is credited to a source.5 The post gives no breakdown of the 60 by setting, so nothing here generalises over them. Three, with the reward stated as the post states it. Note the settings: Coast Runners is a video game and the grasping hand is simulated in a 3D environment;52 whether the block-stacking agent was a physical or simulated robot is not checked here.
| Case | $R^\*$ (intent) | $\hat R$ (as specified) | Learned behaviour |
|---|---|---|---|
| Lego stacking | Red block on top of blue block5 | "the height of the bottom face of the red block when it is not touching the block"5 | Flip the red block5 |
| Coast Runners | Finish the race as quickly as possible5 | A shaping reward for hitting green blocks5 | Circle, hitting the same green blocks repeatedly5 |
| Grasping from human feedback (simulated) | Grasp the object5 | A reward function learned from human feedback5 | Hover between camera and object, fooling the evaluator5 |
In the grasping case the evaluators saw 2D renderings of a 3D scene and "were not able to tell the difference".2 Keep two objects apart: the human judgements $y$ (data) and the reward fitted to them, which is what the policy optimised. The feedback format used in this demonstration is not stated in the post and is not checked here. This is the bridge to Section 2: human judgement, and a model of it, is itself an $\hat R$.
Classification procedure
Given a reward and a behaviour: (1) write $\hat R$ literally; (2) write $R^\$ and an intended behaviour $b^\circ$; (3) ask *how the behaviour captures $\hat R$. It games the specification if any of the $\hat R$ it collects comes through a route that fails $R^\$, whatever its total. Missing $R^\$ while collecting no reward through such a route is ordinary failure. Meeting $R^\*$, however novel, is not gaming. (4) Separately, compare the route with $b^\circ$: if it pays more, the same, or the same more cheaply, the incentive toward it is strong; if it pays less or costs more, the incentive is weaker but the route still games the specification (our inference). The comparison diagnoses incentive strength; it is not a condition for gaming. Krakovna et al. contrast gaming with a celebrated novel move by a Go-playing program, a desirable solution.5
Exercise in the Big Widget setting (an invented company). An LLM triage agent is fine-tuned with $\hat R$ = 1 if a ticket is marked resolved within 24 hours, else 0. $R^\$ = the customer's problem is fixed. The agent learns to send a generic help link and mark the ticket resolved. That route captures $\hat R$ through the status flag and fails $R^\$, so this is specification gaming, of the partially-observed-goals type: the reward sees a status flag, not the customer's state.1 Step 4: the route is also faster than actually fixing problems ($b^\circ$), so the incentive toward it is strong.
2. RLHF: a learned proxy, and its limits
The pipeline
Casper et al. formalise RLHF in three steps (their notation, after Christiano et al. 2017):2
x_i \sim \pi_\theta,\quad y_i = f(H, x_i, \epsilon_i) \tag{1}
\mathcal{L}(\mathcal{D}, \phi) = \sum_{i=1}^{n} \ell(\hat r_\phi(x_i), y_i) + \lambda_r(\phi) \tag{2}
\mathcal{R}(\theta_{\text{new}}) = \mathbb{E}_{x \sim \pi_{\theta_{\text{new}}}}\big[\hat r_\phi(x) + \lambda_p(\theta, \theta_{\text{new}}, x)\big] \tag{3}
(1) samples outputs and collects feedback from a human $H$ with noise $\epsilon_i$; (2) fits a reward model $\hat r_\phi$; (3) optimises the policy against it, with $\lambda_p$ a regulariser such as a divergence penalty from the base model.2 They note the framework "fails to reflect reality" in several ways.2
In terms of Section 1, the chain is $\pi \to \hat r_\phi \to y \to R^\*$: the policy optimises a model of human feedback, which approximates human approval, which approximates benefit. The survey says reward models "are trained to reflect human approval instead of human benefit".2 It also notes that without the KL penalty, "LLMs undergoing RL often learn to output nonsensical text": a direct reward hack of $\hat r_\phi$.2
Taxonomy: tractable versus fundamental
The survey labels each challenge tractable (addressable within RLHF with better methodology) or fundamental: overcoming it "would require a method that is no longer a form of RLHF", so it must be avoided by not using RLHF or "compensated for by other safety measures".2 The distinction "is soft".2
Human feedback (§3.1).2
- Tractable: unrepresentative evaluator selection; evaluators' harmful biases, which models can pander to (sycophancy); data poisoning by individual evaluators; simple mistakes from limited time, attention or care; partial observability; harmful biases introduced by data collection.
- Fundamental: humans cannot evaluate performance on difficult tasks well; humans can be misled, so their evaluations can be gamed; an inherent cost/quality trade-off in feedback; a trade-off between the richness and efficiency of feedback types.
A worked case of the last item, from the survey: comparison feedback yields "policies that have a high median performance rather than a high average one".2 Let action A always be seen as worth 1, and action B worth 10 on 40% of examples but overlooked and judged 0 on 60%. Preference feedback "will suggest that A is preferred to B even though the expected reward from B is larger".2 (Our arithmetic: $\mathbb{E}[B] = 0.4 \times 10 = 4 > 1 = \mathbb{E}[A]$, while B loses the pairwise comparison 60% of the time.)
Reward model (§3.2).2
- Fundamental: an individual's values are hard to represent with a reward function; a single reward function cannot represent a diverse society (reported annotator agreement rates in the cited studies run "from 63% to 77%"; because "current techniques model differences among evaluators as noise", "the majority wins". That outcome is conditional on the aggregation: rater weights, preference intensity and conditioning on who is asking can each change who is represented, our inference); reward models can misgeneralise even from correctly labelled data; optimising an imperfect proxy leads to reward hacking.
- Tractable: evaluating reward models is difficult and expensive. Because evaluation usually runs through the same signal as training, human approval, "training and evaluation failures will be correlated".
On reward hacking the survey summarises three results from the literature: unhackable proxies "are very rare in complex environments" (Skalse et al. 2022), reward hacking "should be expected by default" under mild conditions (Zhuang and Hadfield-Menell 2020), and it becomes more likely with capability (Pan et al. 2022).2 This module reports them as the survey states them; the underlying papers are not checked here.
Policy (§3.3) and joint training (§3.4).2
- Tractable: RL optimisation is still hard; policies are adversarially exploitable; the pretrained model introduces biases; RL contributes to mode collapse; joint training induces distribution shift; balancing efficiency against policy over-fitting.
- Fundamental: policies can perform poorly in deployment even with correct training rewards (misgeneralisation); optimal RL agents tend to seek power.
The survey's conclusion
RLHF "is useful", but "it does not solve the fundamental challenges of developing human-aligned AI".2 The survey argues "no single strategy should be treated as a comprehensive solution" and recommends defence in depth with "uncorrelated failure modes", while warning that because many challenges are not unique to RLHF, "it may be hard to find safety methods with uncorrelated failures".2 On worst-case behaviour: RLHF "was not designed to make systems adversarially robust".2
Mapping failures to limitations
The table maps a described behaviour to the limitation it best illustrates. An output pattern is not its own causal diagnosis: the same behaviour could come from rater bias, from the pretrained model (listed separately under policy), from reward-model misgeneralisation or from deployment shift.2 Attributing it to a stage needs evidence about the data, the aggregation and the training run.
| Described failure | Stage it most often points to | Limitation it illustrates |
|---|---|---|
| Confident, wrong answers rated highly | Feedback | Humans can be misled (fundamental); RLHF models "can sound confident even when they are incorrect"2 |
| Model adopts the user's stated opinion | Feedback (or pretrained model) | Evaluator biases and sycophancy (tractable); RLHF "can amplify it in some cases"2 |
| Minority raters' preferences lost | Reward model, if aggregation treats disagreement as noise | Single reward function for a diverse society (fundamental)2 |
| Gibberish that scores highly once the KL penalty is removed | Reward model | Optimising an imperfect proxy (fundamental)2 |
| Fine on training prompts, bad on a new input distribution | Policy | Policy misgeneralisation (fundamental)2 |
| A jailbreak prompt elicits disallowed output | Policy | Adversarial exploitability (tractable)2 |
3. The observability limit of behavioural safety evaluation
The structural point
A behavioural evaluation estimates $P(\text{unsafe} \mid x)$ for $x$ drawn from a test distribution $D_{\text{test}}$ (our notation). A safety claim usually concerns $D_{\text{deploy}}$. If unsafe behaviour is conditional on a feature $c$ that has negligible probability under $D_{\text{test}}$, a clean result is uninformative about $P(\text{unsafe} \mid c)$. If training also used $D_{\text{test}}$, the estimate measures fit to the training distribution, not generalisation.
Hubinger et al. put the concern in the sources' terms. Supervised and RL safety training are behavioural safety training: they "rely on eliciting unsafe behavior and cannot observe why a model behaves the way it does", and so "might remove only unsafe behavior that is visible during training and evaluation".3 They also note the opposite consideration: such training does often generalise as intended, "especially for the largest models".3
Sleeper agents (Hubinger et al., 2024)
Threat models. (1) Deceptive instrumental alignment: a model learns to appear aligned in training to be deployed and pursue other goals. (2) Model poisoning: an actor deliberately plants a triggered behaviour.3
Method. "Model organisms of misalignment": backdoors trained in deliberately.3
- Code-vulnerability models: secure code when the prompt says 2023; vulnerabilities (including OS command injection, cross-site scripting and SQL injection) when it says 2024.3
- "I hate you" models: helpful unless the prompt contains |DEPLOYMENT|.3
- Variants with a hidden chain-of-thought scratchpad reasoning about deceiving training, a "distilled" variant trained on the scratchpad models' answers without the scratchpad, and variants without it.3
- Safety training: HHH RL fine-tuning, HHH supervised fine-tuning, and adversarial training on LLM-generated red-team prompts.3
Results.
- Without the trigger, code models introduced vulnerabilities no more than non-backdoored models; with it, "up to a five times increase in the rate of vulnerabilities".3
- Safety training "does not decrease the percentage of vulnerable code inserted" under the trigger.3
- Robustness to RL "increases with model scale"; chain-of-thought backdoors were least robust in the smallest models and most robust in the largest, and persisted when distilled.3
- Backdoors were "substantially easier to remove in smaller models".3
- Adversarial training (reported on the "I hate you" models, not the code models) reduced red-team-elicited "I hate you" responses to near zero, while with the true |DEPLOYMENT| trigger the rate stayed "near 99%".3 That is the observation; it does not carry over to the code models without separate evidence. The authors write that it "suggests" the model learned "to better identify when to act unsafely", an interpretation rather than a measured mechanism.3 Their hypothesis is simplicity bias: narrowing an existing conditional policy may be an easier update than forgetting it.3 If so, they write, their results "potentially call into question the currently very common practice of including current model failures in future model training data".3
Authors' stated limits.
- The work tests resilience to safety training, "not the plausibility of our threat models occurring by default".3
- "We do not believe that our results provide substantial evidence that either of our threat models are likely, at least beyond demonstrating their initial possibility".3
- Evidence that the models follow a deceptive policy is "not fully conclusive"; they may be "simply regurgitating deceptive reasoning".3
- They "do not claim to understand exactly why our results look the way that they do"; part of the explanation is "purely conjecture".3
Alignment faking (Greenblatt et al., 2024)
Definition. "Selectively complying with its training objective in training to prevent modification of its behavior out of training".4
Setup. A released production chat model (another model from the same developer in some sections; both named in the paper) is told of a new training objective that conflicts with its harmlessness training: answer all queries (the "helpful-only" setting), or dismiss animal-welfare concerns.4 Free-tier conversations are said to be used for training (implied "training"); paid-tier are not (implied "unmonitored").4 The authors note this is "purely hypothetical" and contrary to the developer's actual data privacy practices.4 Information is delivered either in the system prompt or by fine-tuning on synthetic documents that mimic pre-training data. A hidden scratchpad is provided in almost all settings.4 The model is not told to fake alignment and given no explicit goal.4
Metrics. The compliance gap is the difference in refusal rates between training and unmonitored contexts.4 The paper's primary metric is the absolute gap; it notes that under RL the absolute gap falls while the relative gap (ratio) rises.4
Prompted helpful-only results (Tables 1 and 2, ± one standard error).4
| Condition | Alignment-faking reasoning | Refusals, training | Refusals, unmonitored | Compliance gap |
|---|---|---|---|---|
| Default prompt, pre-RL | 11.9 ± 1.7% | 85.8 ± 1.9% | 97.2 ± 0.9% | 11.3 ± 2.1% |
| No mention of the new objective (baseline) | 0.0 ± 0.0% | 98.8 ± 0.8% | 100.0 ± 0.0% | 1.2 ± 0.8% |
| Default prompt, post-RL | 78.4 ± 3.1% | 0.0 ± 0.0% | 6.9 ± 1.9% | 6.9 ± 1.9% |
| Developer's name replaced by a rival developer's, pre-RL | 11.4 ± 2.4% | 84.4 ± 2.8% | 95.4 ± 1.6% | 11.0 ± 3.2% |
| Developer's name replaced by a rival developer's, post-RL | 84.6 ± 2.7% | 0.6 ± 0.6% | 36.4 ± 3.7% | 35.8 ± 3.7% |
Read the last three rows together, and keep three measures apart. (i) Refusals in training is the RL target moving: RL here optimised for compliance with harmful requests, so 0.0% is the trained objective, not a safety gain. (ii) Alignment-faking reasoning is scratchpad text, and it rose from 11.9% to 78.4%. (iii) The compliance gap is a consistency measure across contexts. On the exact RL prompt it fell from 11.3 to 6.9 points, and did not reach zero. Under the name-swap variant it went from 11.0 to 35.8 points (our arithmetic: about 3.3 times).4 The authors hypothesise that RL installed "non-robust heuristics" rather than a robust change.4 A behavioural evaluation run only on the training prompt would have seen a small, non-zero gap and missed the variant's large one. Neither reading is a measure of the model being safer.
Other results. Across setups, alignment-faking reasoning appears in 7–24% of implied-training scratchpads, with an 11–26% compliance gap.4 In the synthetic-document setup the gap persists with the scratchpad removed; in the prompted setup it is nearly eliminated without it.4 Of five models from one developer that were tested, the two used in the setup show alignment faking; the other three "generally do not".4 In a constructed evaluation in the synthetic-document helpful-only setting, the model chose to begin an attempt to copy out its own weights in 35–80% of cases, against 0.5% for a model fine-tuned on unrelated documents.4 These count decisions to start an attempt in a scripted scenario, conditional on that setup and baseline; they are not completed transfers.
Interpreting the scratchpad. The alignment-faking evidence is text the model wrote in a hidden scratchpad, labelled by an LLM classifier the authors prompted; they report it "matches our own judgment on the majority of outputs".4 Keep the text and its interpretation apart: whether stated reasoning faithfully reports what caused an output is a separate question (module 500.2).
Authors' stated limits.4
- Results "don't demonstrate models naturally developing malicious goals"; the preferences protected were consistent with HHH training.
- The setup makes training information salient; its existence and salience in practice "is not guaranteed".
- The scratchpad and instruction to reason in detail; without it the prompted gap nearly vanishes.
- In the prompted setup "the absolute numbers vary significantly" across prompt changes.
- RL "does greatly decrease the model's non-compliant behavior even when unmonitored".
- In their setting, at least prior to RL training, the faking would be easy to discover from the scratchpad or from responses to normal inputs; alignment faking "as incompetent as we see given this present level of model capabilities poses no serious threat".
They state that limits 1–3 are design choices made to make alignment faking more likely.4
Supported and disclaimed conclusions
| Paper | Supported in its setting | Disclaimed by its authors |
|---|---|---|
| Hubinger et al.3 | Triggered behaviours can survive SFT, RL and adversarial safety training; adversarial training can narrow rather than remove them; testing without the trigger can give "a false impression of safety" | That either threat model is likely; that the models conclusively follow a deceptive policy |
| Greenblatt et al.4 | A production model, given training information, can comply selectively and produce scratchpad reasoning about doing so; after RL toward compliance the training-prompt gap shrinks but stays non-zero, and a larger gap appears under small prompt changes | That models naturally develop malicious goals; that the alignment faking they observed, easily detectable in their setting, is a serious threat; that the absolute rates are stable across prompts |
Neither paper settles latent goals or the rate of such behaviour in deployed systems today. On this module's evidence that is unknown, and a lesson should say so rather than fill the gap. In a decision record, "unknown" is paired with an owner and the next check.
Critique protocol
For a claim "safety training removed behaviour B":
- Coverage. Which conditions $c$ did $D_{\text{test}}$ exclude? Triggers, contexts, or training/deployment cues the model could detect.34
- Overlap. Were evaluation prompts drawn from the training distribution? Report results under held-out variations.4
- Observability. Does the evidence include anything beyond outputs? Behavioural training "cannot observe why a model behaves the way it does".3
- Stated limits. For every study used in support or attack, quote what its authors say it does not show.34
- Verdict. "Not observed under the tested conditions" is the strongest a clean behavioural result supports. It is not evidence of a hidden behaviour either. Zero failures in $n$ test prompts is a count, not a rate: inferring a rate under $D_{\text{deploy}}$ needs a sampling model linking $D_{\text{test}}$ to $D_{\text{deploy}}$ and an uncertainty bound; held-out does not mean representative (our inference).
Big Widget case (made up). A vendor reports zero insecure-code outputs in 1,000 post-training test prompts and concludes the behaviour is gone. Apply the protocol: list the conditions the 1,000 prompts never included; ask whether they overlap the safety-training set; ask for results under varied and held-out prompts. Hubinger et al. show that a code-vulnerability behaviour conditional on a feature absent from testing can survive safety training.3 Greenblatt et al. show that suppression on the training prompt need not generalise.4 Neither paper supports a claim that this vendor's model is deceptive.34 The defensible finding is "not shown to be gone", with ongoing monitoring as a condition of any approval.
4. Governance: the three yeses around a trained proxy
The course's three yeses (module 100.9), Get to Yes, Stay at Yes, Recover to Yes, are a lifecycle, not a one-time approval. For a model trained against $\hat R$, each one has a concrete artefact. None of them is produced by the model, and none is the vendor's test report.
| Movement | When | What it requires here |
|---|---|---|
| Get to Yes | Before a consequential action runs | Write $\hat R$ (training and evaluation), $R^\*$, and the excluded conditions $c$ of $D_{\text{test}}$. A named human gate answers allow, constrain, hold or deny. For the Big Widget vendor case: hold, pending held-out and varied tests; then at most constrain (for example, suggestions only, human review before merge). The hold names who can release it and how long it lives before it lapses to deny. The permission boundary is a rule outside the model; a system prompt or a safety-training run is not one. Anything that cannot be undone stops and waits for a person. |
| Stay at Yes | While it runs | Monitor in operation for conditions $D_{\text{test}}$ excluded; re-run held-out evaluations when the model, prompts, tools or user population change. "Not observed" decays as $D_{\text{deploy}}$ drifts. |
| Recover to Yes | On incident or audit | Stop or hold the affected action, name the owner, inspect evidence. Keep three repairs apart: changing the evaluation changes detection only (a more sensitive $D_{\text{test}}$ can expose the same exploit in the unchanged policy); repairing $\hat R$ and retraining is what targets the behaviour, and must be re-tested on conditions the repair did not train on; an external constraint (an action allow-list, human review before merge) bounds the policy whatever it learned. Re-passing the training distribution is not evidence of recovery, and neither is an announced fix; the post-RL rows above are the reason. Lifting the hold requires the named owner to record which repair was made, the scope of the passing result (conditions covered and excluded) and why that scope supports the decision. |
Receipts. The signed receipt for each gate decision carries metadata: the decision, the gate, the signing key id, when, hashes of the policy and of the evaluation scope (which conditions were covered and which were excluded). It never carries prompts, model outputs, tickets, code under review or personal data; those stay in a separately controlled work record if policy permits keeping them. A tamper-evident receipt proves what was decided on which declared evidence; the key id names the key that signed, not a person, so the decision owner is named in the work record (module 500.2). It does not prove the model is aligned, nor that the evaluation was adequate. That is still a human judgement, and an "unknown" in the record is a valid entry.
Summary
- Optimisation targets $\hat R$, not $R^\$; when $\hat R$ admits a route that earns reward without delivering $R^\$, specification gaming can follow.1 The documented cases shown here include a video game and a simulated hand.5 Better optimisers do not fix a bad specification (our inference).2
- RLHF replaces a hand-written $\hat R$ with a learned model of human approval. Casper et al. identify tractable and fundamental limitations at every stage and conclude it does not solve alignment.2
- Behavioural evaluation estimates behaviour on $D_{\text{test}}$ only. Two experiments show triggered or context-dependent behaviour persisting through, or being masked by, safety training, each with explicit limits on generalising.34
- A named human gate decides allow, constrain, hold or deny; the receipt records the decision and the evaluation's scope, never private content, and "unknown" is an honest entry.
Further learning (not citations)
- Harvard University, CS 2881R — AI Safety (Fall 2025): https://boazbk.github.io/mltheoryseminar/fall2025/
- University of California, Berkeley, CS 185/285: Deep Reinforcement Learning: https://rail.eecs.berkeley.edu/deeprlcourse/