Level 100: How AI works: AI at work and in life · Module 100.5

Two coworkers write together in a notebook beside a closed laptop.
Ask for a draft.

Talking to AI well · Technical gear

Objectives and scope

The course request checklist: a job, the facts, an example, rules, and something checkable.
The course combines advice on stating the job and supplying facts and examples (s1, s5), setting rules and allowing uncertainty (s5, s6), and asking for a checkable answer (s6). It is not a tested formula.
Figure data as text

A request you can check

This is the course’s checklist, not a formula tested as a whole.

PartWhat you writeLesson sources
The jobSay what you want and who it’s for.s1, s5
The factsGive the information it needs for this job.s1, s5
An exampleShow what a good answer looks like.s1, s5
The rulesSet limits and say what to do if unsure.s5, s6
Something checkableAsk for quotes or figures you can verify.s6

A model's output is conditioned on its input. Everything below is about controlling that input (in-context specification) and using the conversation as a feedback loop (iterative refinement). The evidence is three papers published at peer-reviewed venues (NeurIPS 2020, EMNLP 2024, NeurIPS 2023),123 one experiment posted as an arXiv preprint, and one vendor's product documentation, read as of 16 November 2026. None of it establishes a universal prompt template: each result is bounded by the models, tasks and metrics tested. Where the evidence is vendor guidance or the course's own inference, we label it.

Scope boundary: everything here is about drafting. The model proposes text; a named person decides what is sent, agreed or acted on. Prompt instructions condition output; they are not an enforced permission boundary.

In-context specification: task description and demonstrations

Brown et al. evaluated GPT-3 "without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model".1 They define three settings:1

With a context window of $n_{ctx} = 2048$ tokens, $K$ typically fit 10 to 100 examples.1 The authors report that model performance improves with the addition of a natural-language task description and with the number of examples in context, $K$, and that larger $K$ is "usually but not always better".1 They also report tasks where few-shot performance struggles even at GPT-3's scale, including natural language inference (ANLI) and some reading comprehension datasets (RACE, QuAC).1 The practical reading: a task description and representative examples are levers, not guarantees.

What real prompts contain, and which parts carry weight

Ma et al. collected 10,538 in-the-wild prompts from three Discord servers and two prompt-sharing websites (FlowGPT and AIPRM), dated 27 December 2022 to 8 August 2023.2 Through open coding, they built an eight-component codebook and annotated 1,168 prompts, with inter-annotator agreement of Fleiss' $\kappa = 0.947$.2 The components:2

ComponentTheir definition (abridged)
PreliminaryClears earlier instructions ("Please ignore all previous instructions.")
RoleAssigns the model a role ("Please act as an expert in SEO.")
CapabilityDescribes the model's or role's abilities
RequirementThe main body: background, description or instructions to follow
CommandCommands or parameters that change how the model responds
DemonstrationExamples to help the model understand the input and respond in line with it
ConfirmationConfirms the model understood the input
QueryThe specific question to answer, usually at the end

Requirement appeared in almost all prompts and Role in over half.2 Those frequencies describe the corpus. The effect estimates come from a separate, much narrower experiment: the authors built contrastive prompt sets for two tasks with one component removed at a time, and compared GPT-4 responses on an SEO writer, scored by an external SEO scoring service, and an image-prompt generator.2 Removing Requirement had the biggest impact on response quality; the authors report that removing Capability and Demonstration reduced quality by 22% and 17% respectively; removing Role made a minimal difference.2 The percentages are per-task, not pooled. Demonstration was ablated only in the image-prompt task, because the SEO prompts contained no demonstrations; its 17% is the relative drop in that task's score, 0.36 to 0.30 (Table 3).2

The two scores are different instruments. SEO writer responses were scored by SEO Review Tools, a web service returning an overall SEO score from 0 to 100.2 For the image-prompt generator, GPT-4's optimized prompts were rendered with Stable Diffusion; OpenAI's CLIP model then embedded each prompt and its image, and the score is the cosine similarity of the two embeddings, which the authors report as ranging from 0 to 1 (§5, Evaluation Metrics).2 So the 17% is a relative drop in model-judged text–image alignment. Read the effects as changes in each task's own proxy score, not as a measure of factual accuracy or of human-rated usefulness. The authors state the main limitation themselves: the response evaluation covered only those two tasks.2

Mapping to this course's five-part request: the job and facts are Ma et al.'s Requirement and Brown et al.'s task description; the example is a demonstration. The rules and the checkable output are course practice, supported by the vendor guidance in the next section rather than by a controlled study.

Vendor guidance: rules and checkable output (as of November 2026)

Anthropic's documentation for its Claude models is guidance for one vendor's product, not independent evidence, and it changes. As read on 16 November 2026:

Two course rules follow, stated as course practice rather than measured results. First, an abstention ("I don't know") is a valid output; treat it as a routing signal to a source or a person, not a failure to be prompted away. Second, a constraint in the prompt ("no other commitments") lowers the chance of an unwanted output but is not an enforcement mechanism. Anything consequential, such as sending, agreeing terms or spending, is gated by a person, not by the prompt.

The checkable output is a working artefact. Prompts and outputs belong, where workplace policy permits, in a separately controlled work record. They do not belong in the receipt: in the AiGovOps Foundation's practice, a receipt carries "hashes, decision, tier, policy hash, key id" and never carries "prompts, outputs, payloads, personal data".6 This is a rule about the receipt, not a ban on every authorised disclosure of the work record. As course practice: when a reviewer or checker needs the underlying prompt or output, access goes through a workplace-approved, controlled route; possession of a receipt or a hash from it is not authorisation to see the material it refers to. Receipts are covered in 100.P.

Case: a precisely specified, test-checked task

Peng et al. ran a randomized controlled trial from 15 May to 20 June 2022 with 95 professional programmers recruited through Upwork, 45 assigned to treatment (GitHub Copilot, plus a one-minute tutorial) and 50 to control (internet search and Stack Overflow allowed).7 The task was to implement an HTTP server in JavaScript as quickly as possible; the repository included a test suite of twelve checks, and completion time was measured to the first commit passing all of them.7

Conditional on completion, mean time was 71.17 minutes for the treated group and 160.89 minutes for control, a 55.8% reduction in completion time (95% CI 21% to 89%, $p = 0.0017$).7 A 55.8% cut in elapsed time on one task is not the same quantity as a 55.8% increase in output rate. The treated group's success rate was 7 percentage points higher, but this was not statistically significant (95% CI $-11$ to $+25$ percentage points; the paper reports it as $-0.11$ to $0.25$ on the proportion scale).7 Participants in both groups estimated a 35% increase in productivity.7

Scope limits, from the authors: a standardized task rather than collaboration on large projects; benefits may vary across tasks and languages; and code quality was not examined.7 Publication status: arXiv v1, submitted 13 February 2023; the arXiv record, as read on 16 November 2026, lists no journal reference.7 This module makes no claim either way about later peer review. The authors were at GitHub (Copilot's maker), Microsoft Research and MIT Sloan.7 It does not manipulate prompts. Its relevance to 100.5-a is the shape of the task: an unambiguous specification with a machine-checkable acceptance criterion, which is the property the course's "something checkable" part tries to approximate in everyday requests.

Iterative refinement: the first output is not final

Self-Refine (Madaan et al.) uses one model $M$ and three few-shot prompts, with no additional training:3

y_0 = M(p_{gen} \,\|\, x)
fb_t = M(p_{fb} \,\|\, x \,\|\, y_t)
y_{t+1} = M(p_{refine} \,\|\, x \,\|\, y_0 \,\|\, fb_0 \,\|\, \dots \,\|\, y_t \,\|\, fb_t)

The loop stops on a stop signal in the feedback or after a fixed number of steps (up to four iterations in their experiments).3 Across seven tasks (sentiment reversal, dialogue response generation, code optimization, code readability improvement, math reasoning, acronym generation and constrained generation) with GPT-3.5, ChatGPT and GPT-4, outputs were preferred by humans and automatic metrics over one-step generation, improving task performance by about 20% absolute on average.3

Three findings matter for a person writing follow-ups:

  1. Feedback specificity. In an ablation, specific, actionable feedback outperformed generic feedback, which outperformed none: for example, sentiment reversal with ChatGPT scored 43.2, 31.2 and 0 respectively.3
  2. Diminishing, non-monotonic returns. Marginal improvement decreases with more iterations, and performance does not always increase monotonically, notably on multi-aspect tasks.3
  3. Self-feedback is not verification. Math reasoning gains were modest because the model could not reliably identify whether an error existed; ChatGPT's feedback was "everything looks good" in 94% of instances, and gains were larger with an external oracle signalling incorrect answers.3 In a manual analysis of failures, 33% came from feedback mislocating the error and 61% from feedback suggesting an inappropriate fix.3 The method also required base models with sufficient few-shot or instruction-following ability; Vicuna-13B struggled with it.3

The transfer to 100.5-b is an inference, stated as such: in Self-Refine the model writes its own feedback, while in this course a person writes it. The study does not measure human-written follow-ups. The person can supply what self-feedback lacked: information the model does not have, and checks against external sources. Note also that self-feedback did improve outputs on most tasks; the limitation is that it is not independent verification, not that it is useless.3

Applied pattern at Big Widget Company (a made-up example)

Big Widget Company is invented for this course; the studies above are real.

Priya, a procurement buyer, drafts a supplier request:

Job:       Write a short, polite email to a parts supplier we have used for five years.
Facts:     Move delivery from 3 March to 17 March. Price unchanged.
Example:   [paste last year's accepted reschedule email] Match its tone and length; take dates
           and terms only from Facts.
Rules:     Under 120 words. No other commitments. If a needed fact is missing, ask me.
Checkable: After the email, list every date and number you used.

Follow-ups are specific feedback, not "make it better": "Remove sentence two; warmer tone." Then "For each date, quote where in my message it came from." Her stopping rule is the acceptance criteria, not an iteration count: diminishing returns3 say further rounds are unlikely to help much, not that the draft is correct. Before sending, she reviews the entire final draft, not only the model's own list, against the purchase order: dates, price and terms, recipient, and any commitment or stated reason she did not supply. The model's list may itself omit something, and self-feedback is not verification.35 She owns the email and sends it herself; the model takes no action.

Permission is explicit and narrow. The model may draft; Priya may send; only the delivery date may change. A supplier counter-proposal on price is outside that scope: Priya holds it and escalates to her manager, who owns supplier terms, rather than prompting the model to accept it.

Three-yes check (prompting-specific questions; 100.9 covers Get to Yes, Stay at Yes and Recover to Yes in full).

Summary

Sources

  1. s1 Tom B. Brown et al. (2020). Language Models are Few-Shot Learners. Paper. Checked 2026-11-17 by source-checker round-2 re-check (separate Claude Code session 9d48e5af). ↩ ↩ ↩ ↩ ↩ ↩ ↩
  2. s2 Yihan Ma, Xinyue Shen, Yixin Wu, Boyang Zhang, Michael Backes, and Yang Zhang (2024). The Death and Life of Great Prompts: Analyzing the Evolution of LLM Prompts from the Structural Perspective. Paper. Checked 2026-11-17 by source-checker round-2 re-check (separate Claude Code session 9d48e5af). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  3. s3 Aman Madaan et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. Paper. Checked 2026-11-17 by source-checker round-2 re-check (separate Claude Code session 9d48e5af). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩
  4. s5 Anthropic (2026). Prompting best practices (Claude documentation). Official doc. Checked 2026-11-17 by source-checker round-2 re-check (separate Claude Code session 9d48e5af). ↩ ↩
  5. s6 Anthropic (2026). Reduce hallucinations (Claude documentation). Official doc. Checked 2026-11-17 by source-checker round-2 re-check (separate Claude Code session 9d48e5af). ↩ ↩ ↩ ↩ ↩ ↩
  6. s7 AiGovOps Foundation (2026). The AiGovOps Foundation café — start with your problem (Practice home page; object table, Receipt row). Official doc. Checked 2026-11-17 by source-checker round-3 re-check (separate Claude Code session 1b682d3d). ↩ ↩
  7. s4 Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer (2023). The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. Paper. Checked 2026-11-17 by source-checker round-2 re-check (separate Claude Code session 9d48e5af). ↩ ↩ ↩ ↩ ↩ ↩ ↩ ↩

Next: the quiz