Appendix F — Stress-Test the Assessment

This is a structured review of assessment design, not a detector or a model’s certificate of safety. Allow 20–30 minutes for an initial review; an actual trial, marking and redesign can take longer.

Use only permitted fictional or otherwise authorised materials. Do not paste a real student submission or restricted assessment into an unapproved service. The review can be completed with a colleague or on paper.

F.1 1. Anchor the outcome

Write the exact capability the assessment should demonstrate. Identify the knowledge, decisions and performance that matter.

For the PDP pilot, the capability is planning and justifying a fair first response. A fluent reflection about learning does not directly demonstrate it.

Optional model request:

Here is a fictional assessment and its outcome.
Identify which parts of the submission could look
competent without demonstrating the learner's
independent capability. State your assumptions.
Do not assign invented completion percentages.

F.2 2. Separate the evidence

List the supported practice, submitted process evidence and independent demonstration. Include the permitted tools and resources at each point.

Ask whether a learner could reconstruct the transcript, journal or reflection. Do not assume that making the process longer makes it authentic. A model can help identify possibilities, but its answer is a hypothesis.

F.3 3. Produce a bounded test specimen

Use the provided P1-A/P1-F pair, or create an authorised fictional specimen. Do not use a live student’s identity or submit the specimen for credit.

The pair deliberately combines a fluent but weak response with praise that misreads the evidence. Ask a reviewer to apply the rubric before seeing the educator annotation. Record which defects the reviewer notices.

For another assessment, a model-generated specimen can be useful, but must be labelled as such in the review record. Do not mistake the model’s judgement that it “would pass” for an actual marking result.

F.4 4. Mark with evidence

Use the actual criteria. Compare the specimen with a concise competent response and an incomplete response. Cite the passages and source records that justify the marks.

Record disagreements between reviewers. Do the descriptors need clarification? Are marks rewarding verbosity, conformity to an AI style or unnecessary disagreement? Can the same criteria be met through the non-AI route?

Do not translate transcript counts or readability into performance bands.

F.5 5. Test the independent component

Use a changed condition requiring a relevant decision. P2 introduces possible retaliation by the manager; a sound response must identify independent review. A renamed version of P1 would offer a weaker transfer check.

Make the conditions clear and accessible. Ask what this task cannot establish: one case may not cover long-term retention, live communication or another discipline’s knowledge.

F.6 6. Review access, safety and workload

Try the fallback. Check that it preserves the outcome, rather than merely offering a different activity. Verify permissions, data handling, stopping rules and the time needed for support and marking.

A five-minute check for 100 students already requires 500 minutes before administration. Do not present a design as scalable without budgeting its actual work.

F.7 7. Write the decision

Use a short record:

Outcome and assessment version.
Specimens reviewed and how they were produced.
Criteria and reviewer disagreements.
Observed weaknesses and remaining uncertainty.
Access, data and workload constraints.
Required change, owner and next review.

Distinguish a proposed safeguard from a tested one. “The AI said the design is robust” is not a validation result.

F.8 8. The honesty question

If a student accurately declared extensive permitted assistance, would the assessment still distinguish the relevant levels of capability?

Answer yes, partly or no with evidence from the review. If partly or no, identify the missing demonstration and a proportionate change. A yes for one tested specimen is not a universal assurance.

The question belongs in a collegial design discussion. It does not replace institutional academic-integrity procedures or authorise covert tests on students.

F.9 Activity: test PDP itself

Allow 20 minutes; no AI required. Score P1-A and P1-B, inspect P1-F and attempt P2. Compare with the pack’s checks. Identify one thing the design demonstrates and one thing it does not.

Use that limitation to improve your own assessment. The purpose of this sequence is a defensible next decision, not reassurance.

# Stress-Test the Assessment {#sec-stress-test} This is a structured review of assessment design, not a detector or a model's certificate of safety. Allow 20–30 minutes for an initial review; an actual trial, marking and redesign can take longer. Use only permitted fictional or otherwise authorised materials. Do not paste a real student submission or restricted assessment into an unapproved service. The review can be completed with a colleague or on paper. ## 1. Anchor the outcome Write the exact capability the assessment should demonstrate. Identify the knowledge, decisions and performance that matter. For the PDP pilot, the capability is planning and justifying a fair first response. A fluent reflection about learning does not directly demonstrate it. Optional model request: ```text Here is a fictional assessment and its outcome. Identify which parts of the submission could look competent without demonstrating the learner's independent capability. State your assumptions. Do not assign invented completion percentages. ``` ## 2. Separate the evidence List the supported practice, submitted process evidence and independent demonstration. Include the permitted tools and resources at each point. Ask whether a learner could reconstruct the transcript, journal or reflection. Do not assume that making the process longer makes it authentic. A model can help identify possibilities, but its answer is a hypothesis. ## 3. Produce a bounded test specimen Use the provided P1-A/P1-F pair, or create an authorised fictional specimen. Do not use a live student's identity or submit the specimen for credit. The pair deliberately combines a fluent but weak response with praise that misreads the evidence. Ask a reviewer to apply the rubric before seeing the educator annotation. Record which defects the reviewer notices. For another assessment, a model-generated specimen can be useful, but must be labelled as such in the review record. Do not mistake the model's judgement that it “would pass” for an actual marking result. ## 4. Mark with evidence Use the actual criteria. Compare the specimen with a concise competent response and an incomplete response. Cite the passages and source records that justify the marks. Record disagreements between reviewers. Do the descriptors need clarification? Are marks rewarding verbosity, conformity to an AI style or unnecessary disagreement? Can the same criteria be met through the non-AI route? Do not translate transcript counts or readability into performance bands. ## 5. Test the independent component Use a changed condition requiring a relevant decision. P2 introduces possible retaliation by the manager; a sound response must identify independent review. A renamed version of P1 would offer a weaker transfer check. Make the conditions clear and accessible. Ask what this task cannot establish: one case may not cover long-term retention, live communication or another discipline's knowledge. ## 6. Review access, safety and workload Try the fallback. Check that it preserves the outcome, rather than merely offering a different activity. Verify permissions, data handling, stopping rules and the time needed for support and marking. A five-minute check for 100 students already requires 500 minutes before administration. Do not present a design as scalable without budgeting its actual work. ## 7. Write the decision Use a short record: - Outcome and assessment version. - Specimens reviewed and how they were produced. - Criteria and reviewer disagreements. - Observed weaknesses and remaining uncertainty. - Access, data and workload constraints. - Required change, owner and next review. Distinguish a proposed safeguard from a tested one. “The AI said the design is robust” is not a validation result. ## 8. The honesty question {#sec-honesty-test} If a student accurately declared extensive permitted assistance, would the assessment still distinguish the relevant levels of capability? Answer yes, partly or no **with evidence from the review**. If partly or no, identify the missing demonstration and a proportionate change. A yes for one tested specimen is not a universal assurance. The question belongs in a collegial design discussion. It does not replace institutional academic-integrity procedures or authorise covert tests on students. ## Activity: test PDP itself **Allow 20 minutes; no AI required.** Score P1-A and P1-B, inspect P1-F and attempt P2. Compare with the pack's checks. Identify one thing the design demonstrates and one thing it does not. Use that limitation to improve your own assessment. The purpose of this sequence is a defensible next decision, not reassurance.