9 Does it actually work?
Suppose you train an image classifier on photographs of green, amber, and red router status lights. Its preview looks convincing. The work changes there: can you defend its use on photographs from a different room, camera, or lighting condition?
Reopen the proof brief you wrote in the Introduction. Record the intended decision, error costs, baseline, acceptance threshold, and held-out test plan before building or inspecting results. This chapter executes that plan; it does not choose the target after seeing the score.
9.1 Choose your route
Paper-based Core, 30 minutes: use the complete fictional counts in the Tessera case below. Calculate per-class and per-condition results, compare them with the stated threshold, and write a decision. A calculator or spreadsheet is enough. This tests your evaluation reasoning, not your ability to train a model.
Build-along, allow 60–90 minutes plus setup: use Teachable Machine’s image-project training and preview. Collect your own harmless teaching images; coloured cards labelled healthy, degraded, and fault can stand in for a router. Record that this is a proxy task, not a validated router detector. No customer equipment or photographs are supplied.
- Define the three labels and write a provisional threshold for each relevant condition.
- Collect at least 20 development images per class across several angles and backgrounds. Separately reserve 10 fresh images per class per lighting condition, from a different capture session. Never put the reserved images in training.
- Train on the development images and test the reserved images through the preview. Record the actual label, predicted label, lighting condition, and model version for each. Keep the counts, including errors.
- Compare with a simple baseline, such as always predicting the most common class. Report its fault-detection performance as well as accuracy.
- After any change prompted by these results, treat this test set as development evidence. Collect a new held-out set for a final claim.
These small counts are an exercise design, not evidence of production reliability. Stay within the browser preview: publishing an app is not required. If the interface or hardware prevents completion, use the paper route and label the build “not attempted.” Tool setup is in Appendix F.
9.2 The demo and the deployment are different objects
A working demo proves one thing: the solution runs. It says nothing about whether Tessera should rely on it, and reliance is the only question that matters once a model touches a customer. The gap between runs and reliable is where applied AI actually lives, and it is not a gap you close with a more impressive model. You close it by evaluating: stating what good enough means, measuring against it honestly, and naming where it fails.
The most expensive failure mode in applied AI is not a buggy model; it is a slick model that nobody ever bothered to break. Evaluation is the discipline of breaking it on purpose, before a real user, or a real adversary, does it for you.
9.3 Three lenses, not one number
When you ask “does it work?”, the first answer people reach for is accuracy: it got 90% right. Accuracy is the obvious lens, and it is the one people stop at, which is exactly why it misleads. Headline accuracy is an assertion, not evidence. The question that turns it into evidence is 90% on which class, at what cost to the others?
A router-status classifier that is 90% accurate overall but misses seven in ten faults is useless for triage, because the fault class is the only one that matters and it is the one being sacrificed to keep the headline up. Look at the class that matters, not the headline. Measure per class, always.
Accuracy alone never settles “does it work?” A working evaluation holds the solution up to three lenses:
- Accuracy: how often is it right, per class, measured on data it has never seen?
- Fairness: does it work equally across the groups it serves? The same model can be accurate overall and quietly wrong for a subgroup: a lighting condition, a region, a hand that is not yours.
- Explainability: could you tell Tessera’s CEO, Henrik Larsson, why it flagged this customer, in words a non-technical person would accept? “The model says so” is not an answer; it is the cap on how much anyone should trust it.
The three lenses are not a checklist to tick. They are three different questions that a single number (accuracy) pretends to answer all at once.
“90% accurate” is the phrase that lets a solution look finished when it isn’t. Overall accuracy is a weighted average, and averages hide the people (and the classes) the model fails. A churn model 90% accurate overall that misses 70% of actual churners is useless for retention; the headline never tells you. The cure is mechanical: never report an overall number without the per-class and per-group breakdown beneath it.
9.4 “Good enough” is a judgement, not a number
The same model can be fine in one place and dangerous in another, at exactly the same accuracy. Fitness for purpose is set by the decision the solution feeds, and the threshold is set by the blast radius of being wrong.
If your router classifier mis-routes a healthy ticket to the wrong queue, a human glances at it and moves it along. Small blast radius; an average solution is tolerable. But if it silently calls a fault healthy, an angry customer waits while their router stays down. That is a large, hard-to-reverse blast radius, and the same 80%-accurate model that was fine for the first job is dangerous for the second.
This is the trust tool from the Introduction in another form. Small blast radius tolerates an average solution; large blast radius demands precision and a human in the loop. So before you measure anything, you write one sentence: “For [decision], good enough means [threshold], because [blast radius].” A model without a stated threshold is a demo, not an evaluation. The threshold is the thing you measure against, and the thing you cannot quietly move after the fact to make a weak result look strong.
9.5 Test to destruction
Most testing asks “does it work on normal inputs?” Adversarial testing asks the opposite: what inputs make it fail? You are not being cruel to your model. You are finding its edges before a real user does.
Every solution has a failure surface. The light is dim, the hand is shaky, and the background is a carpet instead of a desk. A customer photographs the router in a dark cupboard at midnight, because that is when the fault showed up. A demo evaluated only on the happy path has proven nothing about the conditions that will actually bite.
Adversarial testing is a posture, not a technique: take every assumption your model is making and feed it the opposite. For a vision classifier, that is lighting, angle, distance, background, a photograph of nothing at all. For a churn model, it is a customer with missing fields, or one sitting right on the decision boundary where flipping a single input flips the prediction. For a RAG setup, it is a question the documents don’t answer, and watching whether it admits ignorance or invents a confident lie.
Then there is VET, a three-move self-adversarial check you run on your own outputs, borrowed from the companion book:
- Verify. Re-derive the load-bearing claim independently. Asking the model “is this right?” is not verification; the thing that produced the answer cannot be the thing that confirms it. Check it against the data, the source documents, a photograph it has never seen.
- Explain. Explain the result in your own words, including what the test measured, the assumptions it relies on, and who it could fail. An explanation copied from the model does not demonstrate your understanding.
- Test. Feed it the cases most likely to break it. The ones that fail go straight into your report.
VET is the discipline that stops you rubber-stamping your own work.
9.6 The honest report
A report that says “82% overall, but only 61% under dim light, below our 80% threshold on the only class that matters” is a stronger report than one that claims the model works. Finding a real failure, naming it, and saying what a human must do about it is worth more than a flawless demo.
A report that only shows successes has told the reader nothing useful, because every demo can be made to succeed on its own training data. The thing a business cannot buy from a vendor’s deck is the honest map of where the solution breaks, and that map is the practitioner’s actual deliverable. Prove, don’t persuade stops being a slogan the moment you write down a threshold your solution misses and ship the report anyway.
This is a constructed dialogue using the counts below, not a transcript of a measured run.
Draft: “The classifier gets 41 of 60 images right. Most predictions are correct, so it is ready for deployment.”
Review: the practitioner checks the arithmetic against the table, then asks for the fault-class denominator. Only 6 of 20 faults were correctly identified; under dim light it was 0 of 10. The original brief required at least 9 of 10 fault detections in each lighting condition.
Decision: reject unattended routing or closure. The result supports further development and, if separately approved, a shadow exercise whose suggestions cannot affect customers. The dossier retains the original draft, the count table, the failed threshold, and the corrected decision.
9.7 What the measurement earned
Placement follows from what you measured: the evidence determines how much human oversight the deployment needs.
Routing a suggestion to a reviewer and clearing a customer fault are different uses. The latter can leave a customer without help and requires stronger evidence and controls. With the results below, do not allow automatic closure or assume a healthy prediction is safe: a missed fault arrives under the wrong label. Review all suggestions in any approved shadow exercise, not only predictions labelled fault.
🌐 Tessera Case: a checkable evaluation
Time: 25 minutes. Inputs: the complete count table below and the brief requiring at least 9/10 fault detections in each lighting condition. No AI tool is required.
The following counts are invented for this exercise, not measurements from Tessera, Teachable Machine, or a particular model. Each row contains ten distinct held-out examples. An incorrect prediction is either of the other two classes; the table does not specify which. You can calculate recall for each class, but not precision or a complete confusion matrix.
| Actual class | Lighting | Tested | Correct |
|---|---|---|---|
| Healthy | Normal | 10 | 10 |
| Healthy | Dim | 10 | 9 |
| Degraded | Normal | 10 | 9 |
| Degraded | Dim | 10 | 7 |
| Fault | Normal | 10 | 6 |
| Fault | Dim | 10 | 0 |
Work it through. Overall accuracy is 41/60, or 68.3%. Per-class recall is healthy 19/20 (95%), degraded 16/20 (80%), and fault 6/20 (30%). The dim-light result is 16/30 (53.3%), with fault recall 0/10. An always-healthy baseline gets 20/60 (33.3%) overall and detects no faults. Beating that weak baseline is insufficient: the candidate fails the brief’s fault threshold in both lighting conditions.
Your task: verify every denominator, then write a 150-word go/revise/stop memo. State the intended use, the failed criterion, who could be harmed, what remains unknown, and the next test. Do not attribute the failure to a specific training defect without evidence; insufficient lighting coverage is a hypothesis to investigate.
Success: the memo rejects unattended customer decisions, retains the failed results, and distinguishes the observed counts from a claim about future reliability. Zero successes out of ten is a serious warning; it does not establish that the true detection rate is exactly zero. Add this memo to the dossier. Appendix D explains the gates a later candidate must pass.
9.8 Compare before adding autonomy
The evaluation tells you where this candidate fails. The next chapter asks whether a different model or a more inspectable method fits the same brief better. Keep the test conditions and acceptance rules comparable. Only after that comparison will we add tools and actions in Can it do the whole job? A weak component does not become reliable because it sits inside an agent.
The paper evaluation and browser build have different deliverables. If you completed the build and want to go further:
- Self-host the model. Export the
model.jsonand weight-shard.binfiles and serve them from a static host or small server you control, after checking the current export documentation and hosting requirements. Serving an exported model is an optional deployment exercise, not production approval. Account for dependencies, access, retention, and monitoring as well as the model files. - Run a systematic fairness audit. Build a held-out test set split deliberately across the conditions that could break the model: dim light, off-angle, phone-shake, a carpet background, a different hand behind the camera. Compute accuracy per condition and per group. Then quantify the worst-group gap: the largest spread between your best and worst condition. That single number is the honest headline. If it is large, no overall accuracy rescues you, and you have just produced the per-group breakdown the three-lens measurement demands.
These are depth, not the real version. The Core is complete on its own.
9.9 🔍 Go deeper
Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification: Joy Buolamwini & Timnit Gebru (Proceedings of Machine Learning Research, 2018). proceedings.mlr.press/v81/buolamwini18a.html Why read this: A study of fairness failing under real conditions: commercial vision classifiers up to 34% wrong on darker-skinned women while near-perfect on lighter-skinned men, all hidden behind a strong headline accuracy. This is your router classifier’s story, at scale, with consequences.
Model Cards for Model Reporting: Margaret Mitchell et al. (FAT* 2019). arxiv.org/abs/1810.03993 Why read this: A model card forces you to publish intended use, evaluation data, and per-group performance before release, the same discipline your honest report is built on, turned into a standard you can adopt.
People + AI Guidebook: Data Collection + Evaluation: Google PAIR (People + AI Research). pair.withgoogle.com/guidebook-v2/chapters/data-collection/ Why read this: Practitioner-facing guidance on the bit that quietly decides fairness: the data and conditions you evaluated on. Its framing of “disproportionate product failure” is exactly the per-group gap your three-lens measurement is hunting for.
9.10 Close the evaluation record
Before leaving the chapter, exchange your memo with a colleague or perform a separate self-review. Recalculate the counts without looking at the worked answer. Identify the failed threshold and trace the recommendation to it. Keep unresolved questions in the dossier. Asking a model to agree with your decision does not replace that check.