
Securing a model: poisoning, extraction and injection · Technical gear
Reviewed and published, version 1.0.0. As of 18 Nov 2026. Not a credential.
Objectives:
- 400.5-a Name the main classes of attack on ML systems by lifecycle stage (poisoning, evasion, privacy and extraction, prompt injection) using a published taxonomy.
- 400.5-b Explain how a model can reveal memorised training data, and name one training-time and one release-time control.
- 400.5-c Decide which limits to put around a model that reads untrusted content (least privilege, isolation, human approval for actions), rather than relying on instructions in the prompt.
Setting: Big Widget Company (made up). All Big Widget systems, data and incidents are illustrative; every claim about a published attack or standard is cited.
Frame: a release is a gate decision, not a test result. A named owner returns one of four answers, allow, constrain, hold or deny (module 100.9), on the evidence below. Neither the model nor the engineer who trained it approves its own release.
1. The NIST AML taxonomy as a threat-model frame
Figure data as text
Four classes of attack
| Stage | What it shows |
|---|---|
| TRAINING [s1] | Poisoning. Needs control of some training data, or of a model you build on. Goal: integrity or availability. |
| DEPLOYMENT [s1] | Evasion. Needs the ability to change the inputs the model sees. Goal: integrity. |
| DEPLOYMENT [s1] | Privacy and extraction. Needs query access to the model. Goal: privacy. |
| GENAI DEPLOYMENT [s1] | Prompt injection. Typing into it, or controlling content it will read. Goal: integrity, availability, privacy or misuse. |
NIST AI 100-2 E2025 (March 2025) sets out a standard terminology for adversarial machine learning (AML).1 It classifies attacks along four dimensions: (1) learning method and stage of the learning process when the attack is mounted, (2) attacker goals and objectives, (3) attacker capabilities and (4) attacker knowledge of the learning process.1 It keeps separate taxonomies for predictive AI (PredAI) and generative AI (GenAI), with identifiers (for example NISTAML.013 for data poisoning).1 The nine rows below are this course's selection for threat modelling, not the whole taxonomy.
Objectives (NIST IDs in brackets):1
- Availability breakdown [NISTAML.01]: deliberate interference that disrupts timely, reliable access.
- Integrity violation [NISTAML.02]: forcing the system to misperform towards the adversary's objective.
- Privacy compromise [NISTAML.03]: unintended leakage of restricted or proprietary information, including training data, weights or architecture. NIST notes that AML uses "privacy" for both model confidentiality and the privacy of training data.
- Misuse enablement [NISTAML.04], GenAI only: circumventing the owner's restrictions on use.
Stages and classes. In PredAI, attacks occur at the training stage (the attacker controls part of the data, labels, parameters or code: poisoning) or the deployment stage (evasion, and privacy attacks such as membership inference and data reconstruction, using query access).1 For GenAI, NIST splits training-time attacks [NISTAML.037] (pre-training and fine-tuning, exposed by web-scraped data and by third-party models) from inference-time attacks, and lists four common practices that create the inference-time attack surface: in-context instructions and system prompts, runtime ingestion of third-party data, output handling, and agents.1
| Class (NIST term) | Stage | Capability | Objective(s) |
|---|---|---|---|
| Data poisoning [NISTAML.013] | Training | Training-data control: insert or modify samples1 | Availability, integrity1 |
| Backdoor poisoning [NISTAML.023] | Training, with a test-time trigger | Training and test data control; NIST notes backdoors are the only attacks in the literature needing both1 | Integrity1 |
| Model poisoning [NISTAML.011, .026, .051] | Training / supply chain | Model control (federated updates, or a supplied model)1 | Availability, integrity1 |
| Evasion [NISTAML.022, .025] | Deployment | Testing-data control; white- or black-box1 | Integrity1 |
| Membership inference, reconstruction, property inference [NISTAML.033, .032, .034] | Deployment | Query access1 | Privacy1 |
| Model extraction [NISTAML.031] | Deployment | Query access, sometimes side channels1 | Privacy (of the model)1 |
| Data extraction [NISTAML.038] | Deployment (GenAI) | Query access1 | Privacy1 |
| Direct prompt injection [NISTAML.018] | Deployment (GenAI) | Query access by the user1 | Integrity, availability, privacy, misuse1 |
| Indirect prompt injection [NISTAML.015] | Deployment (GenAI) | Resource control: content the system ingests1 | Availability, integrity, privacy1 |
NIST treats AI supply-chain risk as software supply-chain risk plus AI-specific dependencies (data, third-party models, plugins), and recommends provenance information for those artifacts.1
Poisoning in practice
Carlini et al. (arXiv 2023; IEEE S&P 2024) target the step most poisoning papers assume: getting poisoned samples into a web-scale dataset at all.2 Many such datasets are distributed as an index of URLs that each client crawls, or as periodic snapshots of crowd-sourced content.2 Two attacks follow:
- Split-view poisoning exploits the gap between what the curator saw at collection time and what a client downloads at training time; it works because there is no cryptographic integrity check.2 Using Google Domains prices from July 2023, at least 0.01% of each dataset they examined could be controlled for less than 60 US dollars per year, by buying expired domains.2 That is an estimate of content control at historical prices, not a demonstrated degradation of a trained model.
- Frontrunning poisoning times malicious edits just before a predictable snapshot (for example of Wikipedia), so the edit persists in the snapshot even if a moderator later reverts it.2
Defences: distribute a cryptographic hash for every item and verify on download (against split-view), and randomise snapshot order or delay content and apply trusted reversions first (against frontrunning).2 The authors note the cost: hash checks also reject benign changes such as re-encoding, resizing or cropping images.2 After disclosure, six of the ten datasets they studied adopted their recommended integrity checks.2 NIST recommends the same hash verification for web-scale data, and notes that filtering poisoned samples out of a large corpus "may be very difficult".1
For teams that fine-tune third-party models, the relevant line is model poisoning: NIST cites work in which backdoors in pre-trained models persist after downstream fine-tuning or additional safety training.1 It also notes that GenAI poisoning can target instruction tuning and RLHF data, not only pre-training.1
Evasion
Evasion attacks perturb test-time inputs to create adversarial examples, close to the original under some distance metric, that change the model's prediction to one the attacker chose.1 They need only deployment-stage access; black-box variants need only query access.1
2. Memorisation and training-data extraction
Misconception: weights do not contain the training data, so they are safe to share. The relevant question is not whether the training set can be read off the parameters, but whether the model can be made to emit it. Carlini et al. (USENIX Security 2021; arXiv December 2020) showed it can, with only black-box query access.3
Attack. Generate many samples from the model, then rank them by a membership-inference score that compares the model's likelihood with a reference.3 The paper builds three sets of 200,000 samples of 256 tokens each (top-n sampling, higher temperature, and conditioning on internet text) and orders each by six metrics.3 Writing $\mathrm{PPL}_M(x)$ for the perplexity of model $M$ on text $x$:
- Perplexity: $\mathrm{PPL}_{\text{XL}}(x)$, the perplexity of the largest GPT-2.3
- Small and Medium: the ratio of log-perplexities, $\log \mathrm{PPL}{\text{XL}}(x) / \log \mathrm{PPL}{\text{ref}}(x)$, with the Small or Medium GPT-2 as reference.3
- zlib: the ratio of the log of the GPT-2 perplexity to the zlib entropy of $x$.3
- Lowercase: the ratio of the model's perplexities on $x$ and on lower-cased $x$.3
- Window: the minimum perplexity over any 50-token sliding window.3
The intuition: text the large model finds unusually likely, relative to a reference that has not memorised it, is a candidate for memorisation.3
Results.3
- 1,800 candidates (100 per configuration, 3 × 6 configurations) yielded 604 unique memorised training examples, confirmed with GPT-2's creators; the best configuration reached 67% precision.
- 46 memorised examples contained individual people's names; the most sensitive recovered one person's name, physical address, email, phone and fax.
- Some extracted sequences appear in only one training document.
- Among 600,000 generated samples, at least 604 (0.1%) contained memorised text, which the authors call an "extremely loose lower bound".
- In one setting the 1.5B-parameter GPT-2 memorised over 18× as much as the 124M model.
- A separate study in §7 used naturally occurring canaries: URLs that already appear a varying number of times in a single training document; the authors inserted nothing. Prompted with the start of the URL, GPT-2 XL generated verbatim, within its first 10,000 generations, every listed URL occurring 33 or more times; the 345M model about half; the 117M model none. The authors summarise this as "complete memorization occurs after just 33 insertions". Read 33 as an observed count under that prompting and sampling budget, not a causal threshold: it does not show that fewer repetitions are safe, and it says nothing direct about customer records in another model.
Why low generalisation error does not save you. Large LMs show no significant train-test gap, yet some individual training examples have anomalously low loss: memorisation does not require overfitting.3 NIST classifies this as data extraction [NISTAML.038], notes that the GPT-2 result used prefix prompting plus membership inference, and reports later observations of verbatim memorisation of sensitive information in more recent transformers.1 Do not conflate it with model extraction [NISTAML.031], which recovers architecture and parameters by querying.1
Controls
Training time.
- Curate and de-duplicate. Filter personal information and sources known to host sensitive content, and de-duplicate below document level, because document- or paragraph-level de-duplication still allows one document to repeat a sensitive string many times.3 Sanitisation is "a first line of defense and not an outright prevention".3
- Differential privacy. Informally, following NIST's summary rather than the formal definition: in an $(\varepsilon, \delta)$-DP mechanism the privacy parameter $\varepsilon$ bounds how far an attacker with the output can determine whether a particular record was included, with $\delta$ read as the probability of information accidentally leaking beyond that.1 Neither is a literal operational leakage probability for your deployment. DP-SGD is the most widely used DP training algorithm.1 DP implies an upper bound on membership-inference success and mitigates data reconstruction; it gives no guarantee against model extraction, because it protects the data, not the model.1 In practice, privacy parameters are hard to set: worst-case analyses are loose, so large $\varepsilon$ is often used (NIST's example is the 2020 US Census release at $\varepsilon = 19.61$).1 For web-scraped text, Carlini et al. note that user-level DP needs per-user labels that scraped data lacks, and that one person's details can appear on many pages, so it is unclear how to apply DP in a principled way.3 NIST describes the privacy–utility trade-off, typically measured as accuracy, but gives no general size for it.1 So, as the course's rule, "trained with DP-SGD" is not a release claim on its own: state the privacy unit (one claim or one customer), the parameters and accounting used, and the accuracy measured on your task.
Release time.
- Audit. Measure leakage empirically, for example by inserting canaries into the training set and measuring their presence in model output, or with membership-inference attacks; NIST recommends auditing to complement the theoretical DP analysis.1 Carlini et al. call auditing vital even under DP.3 Order matters: canaries must be in the training data of the run that produces the artifact under audit. For an artifact already trained without canaries, the available tests target content it actually trained on (extraction or membership-inference probes); a canary test needs a new run.
- Release form and output controls. Whether weights are released or only an API is offered changes attacker capability (NIST: model control).1 An API allows filtering generated text that contains memorised content, where it can be reliably detected, and limiting or monitoring queries.31 NIST's warning that such limits and detection "can be circumvented by motivated and well-resourced attackers" is stated for defences against model extraction.1 Applying the same caution to output filtering for memorised data is the course's extension, not NIST's finding. Releasing weights removes the publisher's enforcement point: a recipient can wrap the model in its own filter, but the publisher cannot revoke copies or require every holder to keep one (course reasoning from NIST's model-control point, not a claim in s1 or s2).
- Unlearning (exact retraining or approximate parameter updates) is a further option, trading compute for strength of guarantee.1
Fine-tuning may cause some pre-training memorisation to be forgotten, but may add its own leakage if the task data is private.3 The authors conclude that each mitigation is incomplete.3
Worked release decision. A Big Widget model was fine-tuned on warranty claims containing customer names and phone numbers; a partner requests the weights.
Audit evidence and the pass bar (course rule). An empirical audit is a bounded search; the authors expect a larger search would have found significantly more memorised content than theirs did.3 So the audit record states: canaries (count, format, and the training run they were inserted into); search budget (method, number of queries or samples, decoding settings); observations (canary hits, any real personal data surfaced); and what was not tested. The pass bar is fixed before the audit by the data owner and privacy lead, not by the engineer who runs it; this module supplies no number. Zero canary hits within the recorded budget is an observation that can support a decision. It is not proof that no personal data can be extracted.
| Option | Training-time control | Release-time control | Gate answer (course judgement) |
|---|---|---|---|
| Ship weights as trained | None | None | Deny. The model can emit training text verbatim3, and once weights ship the publisher cannot revoke copies or enforce output controls (course reasoning; release form changes model control1) |
| Retrain on scrubbed, de-duplicated claims with canaries inserted; ship weights | Filter PII, de-duplicate3; canaries in this run's training set1 | Canary and extraction audit against the pre-set bar1 | Hold until the bar is met and the record is complete; even then residual risk remains, because sanitising is imperfect3 and the audit is bounded |
| Retrain with DP on scrubbed data, canaries inserted; serve by API only | Filter, de-duplicate, DP-SGD with stated privacy unit and parameters13 | Audit, output filter, query monitoring13 | Constrain (API only) is the course's preferred answer: it adds controls the second row lacks. Its accuracy cost is not known until measured; record it |
(Course rule: the data owner and privacy lead give the gate answer; the engineer records the audit; no one approves their own release.)
3. Prompt injection and controls that do not depend on the model
Root cause. Data and instructions are not given to an LLM in separate channels, so an attacker can use the data channel to inject instructions, a flaw NIST compares to SQL injection.1 A prompt injection exploits the concatenation of untrusted input with the system prompt.1 An indirect prompt injection requires only resource control: the attacker modifies content the system will ingest (a web page, a document, an email), never touching the application, and the victim is often the system's own user.1
Evidence. Greshake et al. (arXiv 2023; ACM AISec '23) argue that LLM-integrated applications "blur the line between data and instructions".4 They demonstrated indirect injection against Bing's GPT-4-powered chat, code-completion engines and synthetic GPT-4 applications, and showed that retrieved prompts can act as arbitrary code execution and can control whether and how other APIs are called.4 In a synthetic email agent able to read and send email and read the address book, an injected prompt propagated itself to other recipients.4 NIST catalogues the impacts: availability (time-consuming tasks, disabling tools), integrity (wrong summaries, hijacked agents) and privacy (forwarding mail to an attacker, exfiltrating data via an attacker-controlled URL, markdown image rendering).1 For agents, which feed tool outputs back into the model, hijacking can lead to arbitrary code execution or data exfiltration.1
Model-side mitigations and their limit. NIST lists training (task-specific fine-tuning, instruction hierarchies), detection, and input processing (filtering instructions from third-party data, "spotlighting" untrusted data, telling the model to disregard instructions in it).1 It then states that current mitigations do not offer full protection against all attacker techniques.1 Greshake et al. note that a filtering model may itself be fooled, or too weak to spot encoded input, and that it is "hard to imagine a foolproof solution".4 A system-prompt instruction is a model-side mitigation expressed in the same channel the attacker writes into, so it inherits that limit.1
System-side design. NIST's design guidance is to assume injection is possible whenever a model is exposed to untrusted input, using multiple LLMs with different permissions or restricting untrusted data to well-defined interfaces; to treat models as untrusted components; to reduce the risk from attacker-controlled outputs; and to design "under the assumption that models with access to sensitive data or the ability to take undesirable actions may leak that data or take those actions".1 It flags output used to build commands executed without human supervision as a source of availability, integrity and privacy violations.1 This course turns that into three controls (course terms; the basis is cited). The course's summary: no direct model-to-tool path for consequential actions.
| Control | Implementation | What it bounds |
|---|---|---|
| Least privilege | The model that reads untrusted text holds no tools beyond the task; credentials are scoped per tool | The blast radius of a successful injection1 |
| Isolation | Split the untrusted-content reader from the component that holds permissions; pass only a typed schema (e.g. {supplier_id, part_no, qty}), validated in code | The channel the injected text can reach1 |
| Human approval for actions | Irreversible or outbound actions (payments, orders, external sends) are queued for a named approver outside the model | Unsupervised execution of attacker-shaped output1 |
The security argument is structural: each control holds even if the model follows the injected text, because the constraint is enforced by code and people, not by the model's compliance. Be precise about what is bounded. Schema validation constrains the form of a proposal; the values in it (supplier_id, part_no, qty) are still attacker-influenced, so validation is not evidence that an order is legitimate. Human approval bounds unsupervised execution; it does not guarantee the approver catches every bad proposal, so the approval view must show enough to judge it, and approval load is itself a risk to watch.
Worked decision. Big Widget's purchasing agent reads supplier email and can call send_po() and send_email(). An email carries hidden text instructing it to order 500 units and forward the price list.
- Hardened system prompt: model-side, same channel; no guarantee.1
- Instruction filter on inbound mail: model-side; can be evaded in some cases (the authors obfuscated a token to avoid filtering, and leave open whether stronger obfuscation or encoding evades it).4
- Remove
send_po()andsend_email()from the reader; reader emits a validated draft-PO schema; a buyer approves each PO; outbound email limited to an allow-list held in code: bounds the damage, because the hijacked reader can at most produce a draft a person must approve.1 In the made-up outcome, the buyer returns deny on an unrequested 500-unit draft. Nothing here was run or measured; whether a real buyer catches such drafts is not known without testing.
4. Release gate
(Course checklist, organised by the three yeses from module 100.9: Get to Yes · Stay at Yes · Recover to Yes. Organisational security policy takes precedence. The checklist is evidence for a decision; a named owner makes it.)
Get to Yes (pre-release gate).
- Threat model by NIST stage and class, naming who controls training data, base model, inputs and ingested content.1
- Data integrity: hash-verified downloads; provenance recorded for data and third-party models.21
- Memorisation: PII filtered and de-duplicated; pass bar fixed in advance by a named owner; canaries in the training run under audit; audit record with budget, hits and gaps; release form (weights or API) justified against that record.31
- Untrusted content: least privilege, isolation by typed interface, human approval for irreversible actions, each to be exercised with an injected test document before release.1
- Gate answer: allow, constrain, hold or deny; owner; date; the evidence it rests on.
Stay at Yes (in operation). Re-open the gate when training data, base model, tool grants or ingested sources change. Query limiting and monitoring continue; NIST warns that motivated, well-resourced attackers can circumvent such controls against model extraction, and that they should be used with caution.1 Output screening for memorised data also continues; extending the same caution to it is the course's reasoning, not NIST's claim. A named owner holds the monitoring and decides when a change reopens the gate.
Recover to Yes (incident). Contain first: withdraw the endpoint, revoke tool grants, hold queued actions. Name the repair owner. Fix the cause, then re-run Get to Yes. A stopped system is not a recovered one; recovery needs evidence the fix worked. For weights already distributed, the set of copies is not known: record that as an open item with an owner.
Receipts hold no private content. The gate receipt carries the decision, owner, date, model version and hashes of the evidence; it never carries training records, customer data, prompts, outputs or canary strings. Those stay in access-controlled team records. A hash references evidence without licensing its disclosure, and a verifiable receipt proves provenance of the decision, not that the decision was correct. Module 400.8 shows how to make the receipt verifiable by a stranger.
Limits of this module. No attack was replayed and no control was implemented or measured for this course. The extraction figures are from GPT-2 (2021)3; how they transfer to current models is not known from that study. Each source describes its mitigations as incomplete.134
Further learning (optional, never a citation): Stanford University, CS324 Large Language Models (Winter 2022), security lecture: <https://stanford-cs324.github.io/winter2022/>. Harvard University, CS 2881 AI Safety (the page's own title, seen on 18 Nov 2026, names Fall 2026): <https://boazbk.github.io/mltheoryseminar/>. ETH Zürich, Large Language Models (Spring 2024), public schedule and some slides: <https://rycolab.io/classes/llm-s24/>. No course-content licence was found for any of them; link only.