Assessment dossier: Level 100, Open Certificate in AI Literacy
Draft, version 0.1, for Bob Rapp and Ken Johnston, co-chairs of the review board. Written in the curriculum architect role. Nothing here changes a lesson, a quiz, a module's status, review.yml or levels/100/board.yml, and nothing here issues a credential. Every new choice is a proposal and is marked "needs the review board co-chairs’ (Bob and Ken) decision" or "needs Bob's decision". The learner-facing version is Earn the Open Certificate in AI Literacy.
This answers recommendation 7 of the 2026-11-20 deep review (PR #93): make the credential route visible and defensible, with an objective-to-evidence map, rubric descriptors and examples, marker calibration, an appeal trail and a documented cut-score rationale.
Sources
Only these are relied on. Course facts come from files in this repository.
- Credential spec and credential plan: names, learner process, record fields, revocation, human-gated issuance.
- 100.P, your rule and receipt: the project, the host's table, the pass rule and the Maya example.
- Level 100 objectives map, signed by Ken Johnston.
- Each Level 100 module's
quiz.jsonandworkbook.md, for the item counts below. - AERA, APA and NCME, Standards for Educational and Psychological Testing (2014), open-access PDF, checked 2026-11-20 for Standards 4.18, 4.20, 4.21, 5.21, 6.9, 11.1 and 11.13 to 11.16. They are used as a guide to good practice. This dossier does not claim the course meets them.
- ISO/IEC 17024 and ANSI/ICE 1100 are not checked (the deep review's requests returned HTTP 403). Nothing here claims accreditation, compliance or personnel-certification status.
1. What the certificate decision means
Standard 11.1 asks for a clear statement of the intended interpretation before a credentialing test is used. Proposal, needs the review board co-chairs’ (Bob and Ken) decision:
A named host marked the holder's Level 100 project as meeting all four parts of the 100.P rubric (rule, receipt, check, recovery), in the rubric version in force on the issue date.
It is not a claim that the holder scored well on every lesson, is competent in a job, is safe to deploy AI, or is who the name says. The project is open-book, unsupervised and done with a second person's review, and identity is not checked. These limits are stated on the learner page.
The Level 100 objectives describe an everyday learner, not an occupation, so Standard 11.13 (content justified by credential-worthy performance in an occupation) is used only as a prompt: the claim is kept as narrow as the evidence.
2. Blueprint: objective, item, rubric
Every Level 100 objective has a workbook exercise tagged with its ID and practice quiz items in its module's quiz.json. Quizzes and workbooks are practice: answers stay in the browser and nothing is recorded, so they are not evidence for the certificate decision. The only scored evidence is the 100.P project.
"Project link" is a proposal for how strongly each objective shows up in the project. Direct: a rubric part checks it. Indirect: a passing project normally uses it, but no rubric part checks it on its own. None: the certificate says nothing about it.
| Objective | Short form | Quiz items | Project link (proposal) |
|---|---|---|---|
| 100.1-a | A model learned patterns; it does not look answers up | 3 | None |
| 100.1-b | Fluent is not true | 3 | Indirect: Check |
| 100.1-c | Sort tasks: AI with a checker, or not alone | 3 | Indirect: Rule |
| 100.2-a | Training and tuning with human feedback | 5 | None |
| 100.2-b | Output reflects training data, gaps and cutoff | 6 | None |
| 100.3-a | One token at a time | 4 | None |
| 100.3-b | Fluent falsehood, and one way to check | 6 | Indirect: Check |
| 100.4-a | Usually good at, usually bad at | 3 | Indirect: Rule |
| 100.4-b | The jagged frontier | 3 | None |
| 100.4-c | Measure AI's effect, do not judge by feel | 4 | None |
| 100.5-a | A request with job, facts, example, rules, checkable ask | 6 | Indirect: Receipt (prompt) |
| 100.5-b | Improve an answer with follow-ups | 3 | None |
| 100.6-a | Closed-book model versus open-book tool | 4 | None |
| 100.6-b | An agent and its three limits | 8 | Indirect: Rule, Recovery |
| 100.7-a | Traffic-light rule for what goes into a tool | 4 | Indirect: Rule, Receipt (private details removed) |
| 100.7-b | Deepfake requests and hidden instructions | 5 | None |
| 100.7-c | Automated decisions about people | 3 | None |
| 100.8-a | Check sources, numbers and sense; a second person | 4 | Direct: Check |
| 100.8-b | Claims with no evidence | 7 | None |
| 100.8-c | Fake reviews are deceptive | 4 | None |
| 100.9-a | The organisation answers for its AI | 4 | Indirect: Receipt (owner) |
| 100.9-b | Write one rule (six parts) | 6 | Direct: Rule |
| 100.9-c | Stay at yes and recover to yes | 6 | Direct: Recovery |
| 100.R-a | One good use, one bad use, the question between | 7 | Indirect: Rule |
| 100.R-b | Someone else's AI mistake can land on you | 4 | None |
| 100.P-a | Bring a rule, receipt and recovery plan | 9 | Direct: all four parts |
So the certificate decision directly checks 4 of the 26 objectives, touches 9 more, and says nothing about 13. That is why the claim in section 1 is about the project, not about the whole level.
A wording gap to fix. CURRICULUM.md, the credential spec and the achievement criteria in apps/credentials/core.py say each module has a "five-question quiz". The Level 100 quizzes have 3 to 16 items per module (124 in all). Needs Bob's decision: change that wording, which also changes the signed TEST fixtures, in a separate PR.
3. The rubric and pass rule
The rubric on the learner page uses the host's table from 100.P, word for word, as the "Meets" descriptor, and adds a "Not yet" descriptor for each part. That follows Standard 4.18 (scoring criteria clear enough to score accurately). Proposal, needs the review board co-chairs’ (Bob and Ken) decision: adopt it as rubric version 0.1, and give each later change a new version number so a record can be read against the version it was marked with.
4. Marker calibration plan
Standards 4.20 and 4.21 ask for a specified way to select, train, qualify and monitor scorers, with training examples at each rubric level and a check of scorer agreement. Standard 6.9 asks for documented quality control. Every number below is a proposal: needs the review board co-chairs’ (Bob and Ken) decision.
Anchor set. Four made-up Big Widget Company projects, each with an agreed mark and the reason:
| Anchor | What it shows | Agreed mark |
|---|---|---|
| A | Maya's finished project from 100.P: summary run, counts checked against the raw log, one wrong count fixed | Meets on all four |
| B | Maya's first try from the learner page: vague "only when", one "otherwise" case, one of three counts checked, "tell someone" | Rule, Check and Recovery not yet; Receipt meets |
| C | A held project: a buyer asks the AI to rank three suppliers; it is held because the price list may be confidential; the receipt says "not run" and what is still unknown | Meets on all four (held work can pass) |
| D | A run project whose shareable copy still has a customer's name and order number | Receipt not yet; the other three meet |
Anchors C and D are new and are drafts until the co-chairs agree their marks.
Before marking alone. Each new host marks the four anchors without seeing the agreed marks, then compares. Proposed bar: all 16 part-marks match. If not, the host talks through each disagreement with Ken or an experienced host and marks the anchors again.
During the pilot. Two hosts mark every pilot project on their own, then compare. Record, per part, whether they agreed. Where they disagree, the co-chairs decide or name a board member to decide. Record the reason. After the pilot, report the share of projects where both hosts reached the same pass or not-yet decision. That is the decision-consistency evidence Standard 11.14 asks for.
After the pilot. Double-mark a sample, proposed as one project in five. Recalibrate on the anchors when the rubric version changes, when a new host joins, or when the double-marked sample shows a part with repeated disagreement. Add a real, de-identified project as an anchor only with the learner's permission.
What gets kept. For each mark: the four part-marks, the rubric version, the host, the date and any appeal outcome. Not the project text, which stays with the learner. Needs Bob's decision: where these marking records are kept, who can see them and for how long. They are separate from the credential record, which must not hold project text.
5. Cut-score rationale
Standard 5.21 asks that the rationale for a cut score be documented. Standard 11.15 asks that rules for combining parts be told to candidates before they are assessed. Standard 11.16 asks that the passing level depend on the performance needed and not be moved to control how many pass.
Proposal, needs the review board co-chairs’ (Bob and Ken) decision: the cut score is "Meets on all four parts". It is conjunctive: no points, no totals, no partial pass.
Why:
- It is already the module's rule. 100.P says the project passes when all four parts of the host's table are true. The certificate should not have a looser rule than the lesson teaches.
- Each part carries one of the three yeses. The rule and receipt are getting to yes, the check is staying at yes, the recovery plan is recovering to yes. A project missing one is missing the point of the level.
- A total would hide a missing part. With points, an excellent receipt could make up for no recovery plan. In practice a missing recovery plan is the failure that hurts people.
- It does not punish caution. Held and denied projects can pass, so the rule never rewards running an action that should not run.
- It is told to learners in advance, on the learner page and in the module.
- It is not set by pass rates. There is no pilot data yet. The pilot's double-marking will show whether the descriptors are clear enough. If they are not, the fix is clearer descriptors and anchors, not a lower bar.
The quizzes have no cut score, because they are practice and nothing about them is recorded.
6. Revise, resubmit and appeal
Proposals on the learner page, each needing the co-chairs’ decision: unlimited tries with no waiting time and a full re-mark of all four parts; an appeal asked within 30 days of the mark; a re-mark by a different host within 14 days; if the two hosts disagree, a decision by the co-chairs or by a board member they name who has not marked the project. The 30 and 14 days are new numbers with no source behind them; they are placeholders for the co-chairs to set. Needs Bob's decision: the contact route for asking for an appeal.
7. Decisions needed
Bob and Ken (review board co-chairs):
- The intended interpretation in section 1.
- Rubric version 0.1: the new "Not yet" descriptors, and versioning.
- The cut score: Meets on all four parts, conjunctive.
- Quizzes and workbooks are practice only, not part of the pass decision.
- Host conflicts: no marking a project you reviewed as second person, or of someone you manage.
- Per-part marks with one next step for each "Not yet".
- Revise and resubmit: unlimited tries, no waiting time, full re-mark.
- Appeals: 30 days to ask, 14 days to re-mark, who decides.
- Calibration: the anchor set (C and D are new), the 16-of-16 qualifying bar, double-marking every pilot project, then one in five.
Bob:
- The appeal contact route.
- Where host marking records are kept, who sees them and for how long.
- Whether one host mark counts for both the Open Certificate and Practice 100.
- Fixing "five-question quiz" in CURRICULUM.md, the credential spec and the TEST criteria text.
- The phase gate for this page: it is a draft, linked from the course and the 100.P module.