
Talking to AI well · Technical gear
Objectives and scope
Figure data as text
A request you can check
This is the course’s checklist, not a formula tested as a whole.
| Part | What you write | Lesson sources |
|---|---|---|
| The job | Say what you want and who it’s for. | s1, s5 |
| The facts | Give the information it needs for this job. | s1, s5 |
| An example | Show what a good answer looks like. | s1, s5 |
| The rules | Set limits and say what to do if unsure. | s5, s6 |
| Something checkable | Ask for quotes or figures you can verify. | s6 |
- 100.5-a. Write a request that states the job, gives the facts, shows an example, sets rules and asks for something checkable.
- 100.5-b. Improve an answer by asking follow-up questions instead of accepting the first reply.
A model's output is conditioned on its input. Everything below is about controlling that input (in-context specification) and using the conversation as a feedback loop (iterative refinement). The evidence is three papers published at peer-reviewed venues (NeurIPS 2020, EMNLP 2024, NeurIPS 2023),123 one experiment posted as an arXiv preprint, and one vendor's product documentation, read as of 16 November 2026. None of it establishes a universal prompt template: each result is bounded by the models, tasks and metrics tested. Where the evidence is vendor guidance or the course's own inference, we label it.
Scope boundary: everything here is about drafting. The model proposes text; a named person decides what is sent, agreed or acted on. Prompt instructions condition output; they are not an enforced permission boundary.
In-context specification: task description and demonstrations
Brown et al. evaluated GPT-3 "without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model".1 They define three settings:1
- Zero-shot: no demonstrations; the model gets only a natural-language instruction describing the task.
- One-shot: one demonstration, in addition to a natural-language description of the task.
- Few-shot: $K$ demonstrations of context and completion, then one final context for the model to complete. No weights are updated.
With a context window of $n_{ctx} = 2048$ tokens, $K$ typically fit 10 to 100 examples.1 The authors report that model performance improves with the addition of a natural-language task description and with the number of examples in context, $K$, and that larger $K$ is "usually but not always better".1 They also report tasks where few-shot performance struggles even at GPT-3's scale, including natural language inference (ANLI) and some reading comprehension datasets (RACE, QuAC).1 The practical reading: a task description and representative examples are levers, not guarantees.
What real prompts contain, and which parts carry weight
Ma et al. collected 10,538 in-the-wild prompts from three Discord servers and two prompt-sharing websites (FlowGPT and AIPRM), dated 27 December 2022 to 8 August 2023.2 Through open coding, they built an eight-component codebook and annotated 1,168 prompts, with inter-annotator agreement of Fleiss' $\kappa = 0.947$.2 The components:2
| Component | Their definition (abridged) |
|---|---|
| Preliminary | Clears earlier instructions ("Please ignore all previous instructions.") |
| Role | Assigns the model a role ("Please act as an expert in SEO.") |
| Capability | Describes the model's or role's abilities |
| Requirement | The main body: background, description or instructions to follow |
| Command | Commands or parameters that change how the model responds |
| Demonstration | Examples to help the model understand the input and respond in line with it |
| Confirmation | Confirms the model understood the input |
| Query | The specific question to answer, usually at the end |
Requirement appeared in almost all prompts and Role in over half.2 Those frequencies describe the corpus. The effect estimates come from a separate, much narrower experiment: the authors built contrastive prompt sets for two tasks with one component removed at a time, and compared GPT-4 responses on an SEO writer, scored by an external SEO scoring service, and an image-prompt generator.2 Removing Requirement had the biggest impact on response quality; the authors report that removing Capability and Demonstration reduced quality by 22% and 17% respectively; removing Role made a minimal difference.2 The percentages are per-task, not pooled. Demonstration was ablated only in the image-prompt task, because the SEO prompts contained no demonstrations; its 17% is the relative drop in that task's score, 0.36 to 0.30 (Table 3).2
The two scores are different instruments. SEO writer responses were scored by SEO Review Tools, a web service returning an overall SEO score from 0 to 100.2 For the image-prompt generator, GPT-4's optimized prompts were rendered with Stable Diffusion; OpenAI's CLIP model then embedded each prompt and its image, and the score is the cosine similarity of the two embeddings, which the authors report as ranging from 0 to 1 (§5, Evaluation Metrics).2 So the 17% is a relative drop in model-judged text–image alignment. Read the effects as changes in each task's own proxy score, not as a measure of factual accuracy or of human-rated usefulness. The authors state the main limitation themselves: the response evaluation covered only those two tasks.2
Mapping to this course's five-part request: the job and facts are Ma et al.'s Requirement and Brown et al.'s task description; the example is a demonstration. The rules and the checkable output are course practice, supported by the vendor guidance in the next section rather than by a controlled study.
Vendor guidance: rules and checkable output (as of November 2026)
Anthropic's documentation for its Claude models is guidance for one vendor's product, not independent evidence, and it changes. As read on 16 November 2026:
- "Being specific about the output you want can help enhance results." Providing the context or motivation behind instructions helps the model "deliver more targeted responses".4
- Examples "are one of the most reliable ways to steer Claude's output format, tone, and structure"; they should be relevant, diverse and structured, and the page suggests 3 to 5.4
- Give the model explicit permission to say "I don't know"; the vendor claims this "can drastically reduce false information".5 This module does not cite an independent measurement of that effect.
- "Verify with citations": make the response auditable by having the model cite quotes and sources for each claim, and have it retract any claim it cannot support with a quote.5
- These techniques "don't eliminate" hallucinations "entirely"; the page says to always validate critical information.5
Two course rules follow, stated as course practice rather than measured results. First, an abstention ("I don't know") is a valid output; treat it as a routing signal to a source or a person, not a failure to be prompted away. Second, a constraint in the prompt ("no other commitments") lowers the chance of an unwanted output but is not an enforcement mechanism. Anything consequential, such as sending, agreeing terms or spending, is gated by a person, not by the prompt.
The checkable output is a working artefact. Prompts and outputs belong, where workplace policy permits, in a separately controlled work record. They do not belong in the receipt: in the AiGovOps Foundation's practice, a receipt carries "hashes, decision, tier, policy hash, key id" and never carries "prompts, outputs, payloads, personal data".6 This is a rule about the receipt, not a ban on every authorised disclosure of the work record. As course practice: when a reviewer or checker needs the underlying prompt or output, access goes through a workplace-approved, controlled route; possession of a receipt or a hash from it is not authorisation to see the material it refers to. Receipts are covered in 100.P.
Case: a precisely specified, test-checked task
Peng et al. ran a randomized controlled trial from 15 May to 20 June 2022 with 95 professional programmers recruited through Upwork, 45 assigned to treatment (GitHub Copilot, plus a one-minute tutorial) and 50 to control (internet search and Stack Overflow allowed).7 The task was to implement an HTTP server in JavaScript as quickly as possible; the repository included a test suite of twelve checks, and completion time was measured to the first commit passing all of them.7
Conditional on completion, mean time was 71.17 minutes for the treated group and 160.89 minutes for control, a 55.8% reduction in completion time (95% CI 21% to 89%, $p = 0.0017$).7 A 55.8% cut in elapsed time on one task is not the same quantity as a 55.8% increase in output rate. The treated group's success rate was 7 percentage points higher, but this was not statistically significant (95% CI $-11$ to $+25$ percentage points; the paper reports it as $-0.11$ to $0.25$ on the proportion scale).7 Participants in both groups estimated a 35% increase in productivity.7
Scope limits, from the authors: a standardized task rather than collaboration on large projects; benefits may vary across tasks and languages; and code quality was not examined.7 Publication status: arXiv v1, submitted 13 February 2023; the arXiv record, as read on 16 November 2026, lists no journal reference.7 This module makes no claim either way about later peer review. The authors were at GitHub (Copilot's maker), Microsoft Research and MIT Sloan.7 It does not manipulate prompts. Its relevance to 100.5-a is the shape of the task: an unambiguous specification with a machine-checkable acceptance criterion, which is the property the course's "something checkable" part tries to approximate in everyday requests.
Iterative refinement: the first output is not final
Self-Refine (Madaan et al.) uses one model $M$ and three few-shot prompts, with no additional training:3
y_0 = M(p_{gen} \,\|\, x)
fb_t = M(p_{fb} \,\|\, x \,\|\, y_t)
y_{t+1} = M(p_{refine} \,\|\, x \,\|\, y_0 \,\|\, fb_0 \,\|\, \dots \,\|\, y_t \,\|\, fb_t)
The loop stops on a stop signal in the feedback or after a fixed number of steps (up to four iterations in their experiments).3 Across seven tasks (sentiment reversal, dialogue response generation, code optimization, code readability improvement, math reasoning, acronym generation and constrained generation) with GPT-3.5, ChatGPT and GPT-4, outputs were preferred by humans and automatic metrics over one-step generation, improving task performance by about 20% absolute on average.3
Three findings matter for a person writing follow-ups:
- Feedback specificity. In an ablation, specific, actionable feedback outperformed generic feedback, which outperformed none: for example, sentiment reversal with ChatGPT scored 43.2, 31.2 and 0 respectively.3
- Diminishing, non-monotonic returns. Marginal improvement decreases with more iterations, and performance does not always increase monotonically, notably on multi-aspect tasks.3
- Self-feedback is not verification. Math reasoning gains were modest because the model could not reliably identify whether an error existed; ChatGPT's feedback was "everything looks good" in 94% of instances, and gains were larger with an external oracle signalling incorrect answers.3 In a manual analysis of failures, 33% came from feedback mislocating the error and 61% from feedback suggesting an inappropriate fix.3 The method also required base models with sufficient few-shot or instruction-following ability; Vicuna-13B struggled with it.3
The transfer to 100.5-b is an inference, stated as such: in Self-Refine the model writes its own feedback, while in this course a person writes it. The study does not measure human-written follow-ups. The person can supply what self-feedback lacked: information the model does not have, and checks against external sources. Note also that self-feedback did improve outputs on most tasks; the limitation is that it is not independent verification, not that it is useless.3
Applied pattern at Big Widget Company (a made-up example)
Big Widget Company is invented for this course; the studies above are real.
Priya, a procurement buyer, drafts a supplier request:
Job: Write a short, polite email to a parts supplier we have used for five years.
Facts: Move delivery from 3 March to 17 March. Price unchanged.
Example: [paste last year's accepted reschedule email] Match its tone and length; take dates
and terms only from Facts.
Rules: Under 120 words. No other commitments. If a needed fact is missing, ask me.
Checkable: After the email, list every date and number you used.
Follow-ups are specific feedback, not "make it better": "Remove sentence two; warmer tone." Then "For each date, quote where in my message it came from." Her stopping rule is the acceptance criteria, not an iteration count: diminishing returns3 say further rounds are unlikely to help much, not that the draft is correct. Before sending, she reviews the entire final draft, not only the model's own list, against the purchase order: dates, price and terms, recipient, and any commitment or stated reason she did not supply. The model's list may itself omit something, and self-feedback is not verification.35 She owns the email and sends it herself; the model takes no action.
Permission is explicit and narrow. The model may draft; Priya may send; only the delivery date may change. A supplier counter-proposal on price is outside that scope: Priya holds it and escalates to her manager, who owns supplier terms, rather than prompting the model to accept it.
Three-yes check (prompting-specific questions; 100.9 covers Get to Yes, Stay at Yes and Recover to Yes in full).
- Get to Yes: state the permitted action and its limits before use (draft only; date change only; Priya sends). Is anything in the prompt confidential (prices, terms)? See 100.7. If it is unknown whether last year's email may be pasted, that is a hold until someone who knows answers. Is a named person checking and owning the output? Priya.
- Stay at Yes: iterate with specific feedback and keep an external check on facts on every draft, not only the early ones.35
- Recover to Yes: if a wrong email is sent, stop, own it, correct it with the supplier, and change the prompt or rule that let it through. The changed rule is a repair plan; recovery is shown by later emails that pass the check, not by the change alone.
Summary
- Task descriptions and demonstrations measurably condition output, within limits.12
- In GPT-4 ablations on two tasks, each with its own proxy score (an SEO service score; CLIP text–image alignment), the Requirement component had the biggest impact and Role made a minimal difference.2
- Vendor guidance (not independent measurement) recommends permission to abstain and citation-based auditability, and says these reduce, not remove, fabrication.5
- Refinement with specific feedback improved outputs, with diminishing and non-monotonic returns; model self-feedback can improve a draft but does not reliably detect errors, so it is not verification.3
- The model drafts; a named person gates any consequential action. Prompts and outputs are work records, not receipts.6