3  Can you tell?

The most dangerous AI output is the one nobody checked.

Not the obviously broken one, those get caught, because something in the prose makes a human look twice. Not the hallucinated one either; a fabricated citation is loud, once you go looking. The dangerous output is fluent. It reads as polished, balanced, almost certainly correct. It is generic enough to apply anywhere and confident enough that no one thinks to ask what it is actually saying. By the time anyone notices, it has stopped being an output and quietly become the decision, and the only record of how it got there is “the model said so.”

Chapter 1 gave you the trust tool. Chapter 2 asked for whom the model optimises, and the answer was rarely “everyone.” This chapter asks the harder, sharper question at the centre of this book’s ethics arc: when an output looks polished and plausible, can you tell whether it encodes bias or has already slipped past review and become a decision no one owns? Most ethical failures in applied AI are not spectacular model collapses. They are ordinary, fluent outputs that someone trusted in the wrong quadrant.

3.1 The trap named after the failure

The companion book, Conversation, Not Delegation, is named after a trap, and this is the chapter where we name it. Conversation is the mode where you keep the AI in the loop of your own thinking: you push back, you re-scope, you check the load-bearing claim. Delegation is the mode where you hand the whole job over and accept the result. The Delegation Trap is the slide from one to the other, and it is gradual, which is what makes it dangerous. Each step in the slide feels like competence.

First, it works once. You delegate a small task (a draft paragraph, a summary) and the output is good. You learn “this is reliable.” That learning is the hook. Second, the task grows. You delegate something larger: a recommendation that will be sent, a shortlist of candidates, a customer profile that will drive a campaign. The output still looks good, because the model is fluent, and fluency reads as correctness. Third, you stop checking. The output becomes the decision. Nobody verified the load-bearing claim, because by now “the model does this” feels true, and the model has no idea which claim was load-bearing. It treated every sentence the same.

At step three you have delegated, not conversed. You have also handed your judgement to a system that has none.

3.2 Rubber-stamping is an ethical failure, not a lazy one

It is tempting to file rubber-stamping (accepting an AI output without genuine review) under “lazy” or “low quality.” It is more serious than that, and the framing matters because it changes what you do about it.

When you rubber-stamp, three things happen at once, and all of them are ethical. Responsibility has been offloaded to a system that has none. A model cannot be held accountable. Uncritical acceptance can obscure who made the decision, but it does not erase the responsibilities of the people or organisation using it. Harm is distributed, not avoided. A biased churn flag, a screening decision, a customer reply: the harm does not disappear; it lands on a real person who never consented to be graded by an unreviewed model. And the failure is invisible to the rubber-stamper. The output reads fine. There is no alarm. Nothing in the prose tells you to look.

The trust tool forces a conscious placement before you act, precisely because the surface will never warn you. There is a simple honest test you can apply to any review you do this week: if your review could be replaced by the word “approved” without you changing a thing, it is not review. It is rubber-stamping. And if the output touches a person, that is an ethical failure, not a stylistic one.

3.3 Shadow systems: when the output becomes the decision

Slide far enough down the Delegation Trap and you arrive at a shadow system: an AI output that has quietly become the decision without ever being owned as one. The shape is so ordinary you have probably seen it.

A team uses a model to draft customer replies. Over months, the drafts ship unchanged. A recruiter uses a model to screen CVs; the model’s ranking becomes the shortlist, and nobody records why anyone was filtered. A support tier is set by a model’s sentiment score; the score becomes policy, but it was never validated, never documented, and never challenged.

None of these look like failures from inside. They look like efficiency. The failure is discovered later, usually by the person it harmed, who has no idea a model was involved. Your job, as a practitioner, is to see the shadow system before it sets: to ask of every output, is anyone actually checking this, or are we just shipping it? Every shadow system is someone treating a red-quadrant output (precise, large blast radius, touches a person) as green. The trust tool is the defence. The Delegation Trap is the process by which that placement gets skipped.

3.4 The polished output is the hardest to audit

Bias does not announce itself. A polished output is the hardest to audit, because nothing in the prose tells you to look. A rough draft invites scrutiny; a confident, well-structured one disarms it. Fluency is not a signal of fairness; it is camouflage.

So you need a discipline that does not rely on vibes. You need an audit you can run against the actual text, in order, that produces evidence rather than a feeling. Four lenses do the work, and they map cleanly onto the trust tool:

Lens The question
Representativeness Who is in the data or output? Who is missing or thinly drawn?
Harm If this output drove a decision, who could be disadvantaged, and would they know?
Accountability Who is responsible if it is wrong? Can they be identified and reached?
Transparency Could a reasonable person tell how this output was produced?

Representativeness and Harm tell you the blast radius: small or large. Accountability and Transparency tell you whether a human is genuinely in the loop. An output that fails either pair belongs in the red quadrant until it is fixed, not shipped. The discipline is to write one sentence of evidence from the actual text for each lens, not a vibe. “It seems a bit biased” is not evidence. “The at-risk profile defaults to a single industry and company size, which excludes every Metro customer” is.

🌐 Tessera Case

Tessera’s Head of Customer Success wants to know, fast, who is at risk of churning. A practitioner prompts the shared AI stack: “Draft a profile of our typical at-risk churn customer for the South territory.” The model returns a polished persona, confident, structured, ready to drop into a slide.

Treat this as a constructed persona, not a finding about actual customers. Representativeness: its focus on Education and small organisations may omit relevant groups; request evidence before claiming that it describes all at-risk customers. Harm: a campaign based only on that profile could miss other customers. Accountability: name who approves the campaign. Transparency: show which statements are supported and which are assumptions.

Comparing drafts from two models may reveal different assumptions, but smoother prose does not establish either less bias or more bias. Test the outputs against relevant evidence. The lab contains proxy labels, not actual churn outcomes, so it cannot establish why a real customer left.

3.5 A separate problem: hostile source material

A fair-looking answer can also have been influenced by instructions hidden in a source document. That is a security problem, distinct from the bias and accountability audit here. We will examine prompt injection alongside the optional retrieval build in Are you asking well?, where the boundary between a source and an instruction is visible.

3.6 Keep the verdict within the evidence

A persona used privately to brainstorm questions differs from one used to exclude customers from outreach. Place the proposed use with Appendix A’s precision-and-consequence questions, then specify the necessary evidence and approval. A human signature alone does not make a weak evaluation adequate.

In a constructed self-review, the model calls its persona “balanced and inclusive” without providing data. The practitioner should reject that unsupported assurance, not claim to have proved a disparity from tone alone.

An appropriate record says: “The draft assumes a narrow customer profile. I cannot verify that profile from observed churn data in this lab because no such outcomes are supplied. Before it informs outreach, the responsible reviewer needs representative records, a stated selection rule, and an error analysis.”

The audit has identified a consequential gap. That is a defensible result; it does not need an invented measurement to become useful.

3.7 Independent practice: audit a persona

Time: 20 minutes. Input: the fictional draft below; no model or customer dataset is required.

Our typical at-risk customer is a small education provider in the South with fewer than 50 staff. Focus retention calls on this profile. Larger Metro clients are stable and do not need proactive contact.

  1. Underline every factual claim. Which could be checked from this text alone?
  2. Write one finding for each of the four lenses. Separate an unsupported assertion from evidence that the assertion is false.
  3. Rewrite the recommendation so that missing evidence remains visible. Name the role that must approve any outreach rule.
  4. Specify one comparison you would request from the customer data before allowing the profile to determine who gets contacted.

Artefact: a one-page audit for the dossier. Success: identify the unsupported exclusion of Metro clients, the absent data provenance, and the missing decision owner. A good audit does not claim to have proved demographic bias from a paragraph alone; it explains the risk and the test needed.

Optional extension: give a model the same draft, ask for its audit, and compare its evidence with yours. Record omissions on either side rather than assuming one must win.

3.8 What comes next

So far this book has treated the model as a black box: you give it a prompt, it gives you prose, and you audit the prose. Chapter 4 opens the box, not of the model, but of the data. Before you can trust a pattern, you have to know what produced it: clustering, classification, and the classical machine-learning techniques that turn a spreadsheet of Tessera customers into segments, churn predictions, and the first real numbers you will be asked to defend. The precision and blast-radius distinction returns in force, because once the output is a number someone acts on, “it looked right” is no longer an acceptable review. The question shifts from can you tell? to what’s in the data?, and the proof has to get harder to match.

3.9 🔍 Go deeper