Level 400: Engineering AI systems · Module 400.5

Two colleagues hold a letter up to the café window light, looking for something hidden in it.
Check what it reads

Securing a model: poisoning, extraction and injection · Technical gear

Reviewed and published, version 1.0.0. As of 18 Nov 2026. Not a credential.

Objectives:

Setting: Big Widget Company (made up). All Big Widget systems, data and incidents are illustrative; every claim about a published attack or standard is cited.

Frame: a release is a gate decision, not a test result. A named owner returns one of four answers, allow, constrain, hold or deny (module 100.9), on the evidence below. Neither the model nor the engineer who trained it approves its own release.

1. The NIST AML taxonomy as a threat-model frame

Poisoning hits training and needs control of some training data or a model you build on. Evasion, privacy and extraction, and prompt injection hit deployment: by changing inputs, by query access, or by typing into a GenAI system or controlling content it reads.
The four-class grouping is the course's, for teaching; NIST's taxonomy has more entries. Stages, attacker needs and goals follow NIST (s1).
Figure data as text

Four classes of attack

StageWhat it shows
TRAINING [s1]Poisoning. Needs control of some training data, or of a model you build on. Goal: integrity or availability.
DEPLOYMENT [s1]Evasion. Needs the ability to change the inputs the model sees. Goal: integrity.
DEPLOYMENT [s1]Privacy and extraction. Needs query access to the model. Goal: privacy.
GENAI DEPLOYMENT [s1]Prompt injection. Typing into it, or controlling content it will read. Goal: integrity, availability, privacy or misuse.

NIST AI 100-2 E2025 (March 2025) sets out a standard terminology for adversarial machine learning (AML).1 It classifies attacks along four dimensions: (1) learning method and stage of the learning process when the attack is mounted, (2) attacker goals and objectives, (3) attacker capabilities and (4) attacker knowledge of the learning process.1 It keeps separate taxonomies for predictive AI (PredAI) and generative AI (GenAI), with identifiers (for example NISTAML.013 for data poisoning).1 The nine rows below are this course's selection for threat modelling, not the whole taxonomy.

Objectives (NIST IDs in brackets):1

Stages and classes. In PredAI, attacks occur at the training stage (the attacker controls part of the data, labels, parameters or code: poisoning) or the deployment stage (evasion, and privacy attacks such as membership inference and data reconstruction, using query access).1 For GenAI, NIST splits training-time attacks [NISTAML.037] (pre-training and fine-tuning, exposed by web-scraped data and by third-party models) from inference-time attacks, and lists four common practices that create the inference-time attack surface: in-context instructions and system prompts, runtime ingestion of third-party data, output handling, and agents.1

Class (NIST term)StageCapabilityObjective(s)
Data poisoning [NISTAML.013]TrainingTraining-data control: insert or modify samples1Availability, integrity1
Backdoor poisoning [NISTAML.023]Training, with a test-time triggerTraining and test data control; NIST notes backdoors are the only attacks in the literature needing both1Integrity1
Model poisoning [NISTAML.011, .026, .051]Training / supply chainModel control (federated updates, or a supplied model)1Availability, integrity1
Evasion [NISTAML.022, .025]DeploymentTesting-data control; white- or black-box1Integrity1
Membership inference, reconstruction, property inference [NISTAML.033, .032, .034]DeploymentQuery access1Privacy1
Model extraction [NISTAML.031]DeploymentQuery access, sometimes side channels1Privacy (of the model)1
Data extraction [NISTAML.038]Deployment (GenAI)Query access1Privacy1
Direct prompt injection [NISTAML.018]Deployment (GenAI)Query access by the user1Integrity, availability, privacy, misuse1
Indirect prompt injection [NISTAML.015]Deployment (GenAI)Resource control: content the system ingests1Availability, integrity, privacy1

NIST treats AI supply-chain risk as software supply-chain risk plus AI-specific dependencies (data, third-party models, plugins), and recommends provenance information for those artifacts.1

Poisoning in practice

Carlini et al. (arXiv 2023; IEEE S&P 2024) target the step most poisoning papers assume: getting poisoned samples into a web-scale dataset at all.2 Many such datasets are distributed as an index of URLs that each client crawls, or as periodic snapshots of crowd-sourced content.2 Two attacks follow:

Defences: distribute a cryptographic hash for every item and verify on download (against split-view), and randomise snapshot order or delay content and apply trusted reversions first (against frontrunning).2 The authors note the cost: hash checks also reject benign changes such as re-encoding, resizing or cropping images.2 After disclosure, six of the ten datasets they studied adopted their recommended integrity checks.2 NIST recommends the same hash verification for web-scale data, and notes that filtering poisoned samples out of a large corpus "may be very difficult".1

For teams that fine-tune third-party models, the relevant line is model poisoning: NIST cites work in which backdoors in pre-trained models persist after downstream fine-tuning or additional safety training.1 It also notes that GenAI poisoning can target instruction tuning and RLHF data, not only pre-training.1

Evasion

Evasion attacks perturb test-time inputs to create adversarial examples, close to the original under some distance metric, that change the model's prediction to one the attacker chose.1 They need only deployment-stage access; black-box variants need only query access.1

2. Memorisation and training-data extraction

Misconception: weights do not contain the training data, so they are safe to share. The relevant question is not whether the training set can be read off the parameters, but whether the model can be made to emit it. Carlini et al. (USENIX Security 2021; arXiv December 2020) showed it can, with only black-box query access.3

Attack. Generate many samples from the model, then rank them by a membership-inference score that compares the model's likelihood with a reference.3 The paper builds three sets of 200,000 samples of 256 tokens each (top-n sampling, higher temperature, and conditioning on internet text) and orders each by six metrics.3 Writing $\mathrm{PPL}_M(x)$ for the perplexity of model $M$ on text $x$:

The intuition: text the large model finds unusually likely, relative to a reference that has not memorised it, is a candidate for memorisation.3

Results.3

Why low generalisation error does not save you. Large LMs show no significant train-test gap, yet some individual training examples have anomalously low loss: memorisation does not require overfitting.3 NIST classifies this as data extraction [NISTAML.038], notes that the GPT-2 result used prefix prompting plus membership inference, and reports later observations of verbatim memorisation of sensitive information in more recent transformers.1 Do not conflate it with model extraction [NISTAML.031], which recovers architecture and parameters by querying.1

Controls

Training time.

Release time.

Fine-tuning may cause some pre-training memorisation to be forgotten, but may add its own leakage if the task data is private.3 The authors conclude that each mitigation is incomplete.3

Worked release decision. A Big Widget model was fine-tuned on warranty claims containing customer names and phone numbers; a partner requests the weights.

Audit evidence and the pass bar (course rule). An empirical audit is a bounded search; the authors expect a larger search would have found significantly more memorised content than theirs did.3 So the audit record states: canaries (count, format, and the training run they were inserted into); search budget (method, number of queries or samples, decoding settings); observations (canary hits, any real personal data surfaced); and what was not tested. The pass bar is fixed before the audit by the data owner and privacy lead, not by the engineer who runs it; this module supplies no number. Zero canary hits within the recorded budget is an observation that can support a decision. It is not proof that no personal data can be extracted.

OptionTraining-time controlRelease-time controlGate answer (course judgement)
Ship weights as trainedNoneNoneDeny. The model can emit training text verbatim3, and once weights ship the publisher cannot revoke copies or enforce output controls (course reasoning; release form changes model control1)
Retrain on scrubbed, de-duplicated claims with canaries inserted; ship weightsFilter PII, de-duplicate3; canaries in this run's training set1Canary and extraction audit against the pre-set bar1Hold until the bar is met and the record is complete; even then residual risk remains, because sanitising is imperfect3 and the audit is bounded
Retrain with DP on scrubbed data, canaries inserted; serve by API onlyFilter, de-duplicate, DP-SGD with stated privacy unit and parameters13Audit, output filter, query monitoring13Constrain (API only) is the course's preferred answer: it adds controls the second row lacks. Its accuracy cost is not known until measured; record it

(Course rule: the data owner and privacy lead give the gate answer; the engineer records the audit; no one approves their own release.)

3. Prompt injection and controls that do not depend on the model

Root cause. Data and instructions are not given to an LLM in separate channels, so an attacker can use the data channel to inject instructions, a flaw NIST compares to SQL injection.1 A prompt injection exploits the concatenation of untrusted input with the system prompt.1 An indirect prompt injection requires only resource control: the attacker modifies content the system will ingest (a web page, a document, an email), never touching the application, and the victim is often the system's own user.1

Evidence. Greshake et al. (arXiv 2023; ACM AISec '23) argue that LLM-integrated applications "blur the line between data and instructions".4 They demonstrated indirect injection against Bing's GPT-4-powered chat, code-completion engines and synthetic GPT-4 applications, and showed that retrieved prompts can act as arbitrary code execution and can control whether and how other APIs are called.4 In a synthetic email agent able to read and send email and read the address book, an injected prompt propagated itself to other recipients.4 NIST catalogues the impacts: availability (time-consuming tasks, disabling tools), integrity (wrong summaries, hijacked agents) and privacy (forwarding mail to an attacker, exfiltrating data via an attacker-controlled URL, markdown image rendering).1 For agents, which feed tool outputs back into the model, hijacking can lead to arbitrary code execution or data exfiltration.1

Model-side mitigations and their limit. NIST lists training (task-specific fine-tuning, instruction hierarchies), detection, and input processing (filtering instructions from third-party data, "spotlighting" untrusted data, telling the model to disregard instructions in it).1 It then states that current mitigations do not offer full protection against all attacker techniques.1 Greshake et al. note that a filtering model may itself be fooled, or too weak to spot encoded input, and that it is "hard to imagine a foolproof solution".4 A system-prompt instruction is a model-side mitigation expressed in the same channel the attacker writes into, so it inherits that limit.1

System-side design. NIST's design guidance is to assume injection is possible whenever a model is exposed to untrusted input, using multiple LLMs with different permissions or restricting untrusted data to well-defined interfaces; to treat models as untrusted components; to reduce the risk from attacker-controlled outputs; and to design "under the assumption that models with access to sensitive data or the ability to take undesirable actions may leak that data or take those actions".1 It flags output used to build commands executed without human supervision as a source of availability, integrity and privacy violations.1 This course turns that into three controls (course terms; the basis is cited). The course's summary: no direct model-to-tool path for consequential actions.

ControlImplementationWhat it bounds
Least privilegeThe model that reads untrusted text holds no tools beyond the task; credentials are scoped per toolThe blast radius of a successful injection1
IsolationSplit the untrusted-content reader from the component that holds permissions; pass only a typed schema (e.g. {supplier_id, part_no, qty}), validated in codeThe channel the injected text can reach1
Human approval for actionsIrreversible or outbound actions (payments, orders, external sends) are queued for a named approver outside the modelUnsupervised execution of attacker-shaped output1

The security argument is structural: each control holds even if the model follows the injected text, because the constraint is enforced by code and people, not by the model's compliance. Be precise about what is bounded. Schema validation constrains the form of a proposal; the values in it (supplier_id, part_no, qty) are still attacker-influenced, so validation is not evidence that an order is legitimate. Human approval bounds unsupervised execution; it does not guarantee the approver catches every bad proposal, so the approval view must show enough to judge it, and approval load is itself a risk to watch.

Worked decision. Big Widget's purchasing agent reads supplier email and can call send_po() and send_email(). An email carries hidden text instructing it to order 500 units and forward the price list.

4. Release gate

(Course checklist, organised by the three yeses from module 100.9: Get to Yes · Stay at Yes · Recover to Yes. Organisational security policy takes precedence. The checklist is evidence for a decision; a named owner makes it.)

Get to Yes (pre-release gate).

  1. Threat model by NIST stage and class, naming who controls training data, base model, inputs and ingested content.1
  2. Data integrity: hash-verified downloads; provenance recorded for data and third-party models.21
  3. Memorisation: PII filtered and de-duplicated; pass bar fixed in advance by a named owner; canaries in the training run under audit; audit record with budget, hits and gaps; release form (weights or API) justified against that record.31
  4. Untrusted content: least privilege, isolation by typed interface, human approval for irreversible actions, each to be exercised with an injected test document before release.1
  5. Gate answer: allow, constrain, hold or deny; owner; date; the evidence it rests on.

Stay at Yes (in operation). Re-open the gate when training data, base model, tool grants or ingested sources change. Query limiting and monitoring continue; NIST warns that motivated, well-resourced attackers can circumvent such controls against model extraction, and that they should be used with caution.1 Output screening for memorised data also continues; extending the same caution to it is the course's reasoning, not NIST's claim. A named owner holds the monitoring and decides when a change reopens the gate.

Recover to Yes (incident). Contain first: withdraw the endpoint, revoke tool grants, hold queued actions. Name the repair owner. Fix the cause, then re-run Get to Yes. A stopped system is not a recovered one; recovery needs evidence the fix worked. For weights already distributed, the set of copies is not known: record that as an open item with an owner.

Receipts hold no private content. The gate receipt carries the decision, owner, date, model version and hashes of the evidence; it never carries training records, customer data, prompts, outputs or canary strings. Those stay in access-controlled team records. A hash references evidence without licensing its disclosure, and a verifiable receipt proves provenance of the decision, not that the decision was correct. Module 400.8 shows how to make the receipt verifiable by a stranger.

Limits of this module. No attack was replayed and no control was implemented or measured for this course. The extraction figures are from GPT-2 (2021)3; how they transfer to current models is not known from that study. Each source describes its mitigations as incomplete.134

Further learning (optional, never a citation): Stanford University, CS324 Large Language Models (Winter 2022), security lecture: <https://stanford-cs324.github.io/winter2022/>. Harvard University, CS 2881 AI Safety (the page's own title, seen on 18 Nov 2026, names Fall 2026): <https://boazbk.github.io/mltheoryseminar/>. ETH Zürich, Large Language Models (Spring 2024), public schedule and some slides: <https://rycolab.io/classes/llm-s24/>. No course-content licence was found for any of them; link only.

Sources

  1. s1 Apostol Vassilev, Alina Oprea, Alie Fordyce, Hyrum Anderson, Xander Davies and Maia Hamin (NIST) (2025). Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2 E2025). Standard. Checked 2026-11-19 by source checker (separate session, Claude Opus 5.5; round 3 re-check, all 164 uses supported; PDF fetched from nvlpubs.nist.gov). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  2. s4 Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas and Florian Tramèr (2023). Poisoning Web-Scale Training Datasets is Practical. Paper. Checked 2026-11-19 by source checker (separate session, Claude Opus 5.5; round 2 re-check, all 23 uses supported; round 3 re-check re-confirmed). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  3. s2 Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea and Colin Raffel (2021). Extracting Training Data from Large Language Models. Paper. Checked 2026-11-19 by source checker (separate session, Claude Opus 5.5; round 2 re-check, all 69 uses supported; round 3 re-check re-confirmed, 73 uses by marker count, all supported). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  4. s3 Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz and Mario Fritz (2023). Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. Paper. Checked 2026-11-19 by source checker (separate session, Claude Opus 5.5; round 2 re-check, all 16 uses supported; round 3 re-check re-confirmed). ↩ ↩ ↩ ↩ ↩ ↩

Next: the quiz