A. The Trust Tool
Use this practitioner checklist to decide how much checking an output needs before its intended use. It does not calculate risk, guarantee safety, or replace legal, security, or professional review. AI failures can arise from data, design, implementation, incentives, and operations as well as misplaced trust.
Place the intended use
Ask two questions:
- How exact must this output be? A private brainstorm tolerates unsuitable suggestions. A refund amount, diagnostic result, or quoted source may need exact checking.
- What happens if it is wrong? Consider who is affected, whether the action can be reversed, how quickly errors will be noticed, and whether anyone can challenge the result.
| Small consequences | Large consequences | |
|---|---|---|
| Approximation is acceptable | Green | Yellow |
| Precision is required | Yellow | Red |
“Average” in the chapters means approximation is acceptable for that use. It is not a claim that a model generates a statistical average of its training documents.
Make the check concrete
Green: inspect whether the output serves the modest purpose. A list of private brainstorming ideas can be useful without treating every idea as a fact.
Yellow: identify and verify every claim the decision depends on. There may be several; “check the load-bearing bit” does not mean checking one convenient sentence and ignoring the rest.
Red: require qualified human authorisation and appropriate external controls before consequential use. If no one can evaluate the result, stop or obtain expertise. A signature from an overloaded or unqualified reviewer is not sufficient.
For high-consequence decisions, this book uses a conservative human-review rule even when approximation is acceptable. The grid is a starting checklist, not permission to bypass an applicable approval process.
Keep risk separate from measured quality
A citation makes a claim easier to check; it does not reduce the harm of acting on a false claim. Better accuracy changes the estimated likelihood of some errors, not necessarily the consequences or required precision. A candidate with a higher score therefore does not automatically move from red to yellow.
Controls can change a use: a suggestion kept in a shadow study, unable to affect a customer, differs from the same suggestion closing a ticket. Record that changed use, the controls enforcing it, and any residual harm. Do not merely recolour the model.
Worked placements
| Output and intended use | Starting placement and check |
|---|---|
| Private brainstorm of possible renewal-risk factors | Green: inspect relevance; do not publish hypotheses as findings |
| Numerical regional comparison in a decision memo | Yellow: recompute counts, name the proxy label and small sample |
| Answer grounded in a support policy | Yellow while drafting: verify the supporting passage and missing information |
| Suggested ticket category in a shadow study | Check against the manual process; no live routing, closing, or delay |
| A classifier output that could close a fault | Red: evaluate missed faults, enforce approval and fallback |
| Customer-facing promise about money or service | Red: authorised policy check and approval before sending |
| A hiring or eligibility recommendation | Red: specialist review, applicable obligations, and a way to challenge errors |
A label of “healthy” can conceal a missed fault. Restricting review to predictions labelled “fault” misses this failure. Likewise, a regional summary can influence decisions about a whole group; aggregation does not make it harmless.
Use the four lenses
For yellow or red uses, Chapter 3 supplies four questions:
- Representativeness: who appears in the data, and who is missing?
- Harm: who could be disadvantaged by this particular use?
- Accountability: who owns the decision and can stop or correct it?
- Transparency: which inputs, rules, sources, and limitations can be inspected?
Record evidence and gaps under each lens. These questions do not disclose a model’s internal reasoning or settle fairness by themselves. Unresolved consequential gaps are reasons to withhold approval.
Place each stage separately, then consider how errors travel between stages. In a classification → extraction → drafting chain, an early label can direct all later work toward the wrong issue. Put a check where it can still prevent that error from becoming an action.
Keep the placement, evidence, owner, and review date in the dossier. Revisit them when the model, sources, data, use, or controls change.