D. From Prototype to Production
The dossier may end with a working prototype, a failed evaluation, or a decision not to build. None is production approval. This appendix describes the evidence needed to move a candidate forward, and the conditions under which it should stay where it is.
This appendix is about the gap between proved it works and keeps working in production. It is the layer the twelve chapters point at but do not build out, because it belongs to the operational lifetime of a system, not to the act of building one.
Three gates, not one launch
| Gate | Evidence needed to proceed | If it is missing |
|---|---|---|
| Prototype to bounded pilot | Defined use, held-out evaluation against the brief, data/security review, named owner, tested fallback, limits on users, actions, duration, and volume | Keep it in development; a separate shadow study may be proposed with no effect on customer decisions |
| Pilot to production | Results from representative pilot conditions, measured review workload, incident/rollback rehearsal, monitoring and access controls, approvals from the responsible teams | Extend, redesign, or stop the pilot; do not quietly widen its scope |
| Continued operation | Scheduled and change-triggered evaluation, monitoring of relevant failures, current ownership and approvals, action on breaches of thresholds | Pause the affected use and revert to the manual process until reviewed |
These are decision gates, not universal numerical standards. Set their thresholds for the specific use and risks. Re-baselining after a model, prompt, source, or workflow change means preserving the old evidence and running a new comparison, not rewriting the old acceptance rule.
The honest boundary
The no-code tools in this book (Orange, Open WebUI, Teachable Machine, an agent framework) can help you build a candidate prototype for evaluation; a failed or paper-only exercise is also a valid dossier outcome. They will not, on their own, run a hardened production system. Engineering is needed during prototyping too; production adds operational demands such as monitoring, rollback, access control, incident response, scaling, cost management.
The dangerous practitioner is the one who ships a polished prototype as if it were a production system, because they cannot see the line. The useful practitioner can say, for any artefact: this far I can build, evaluate, and defend; from here, engineering and operations take it, and here is exactly what they must not drop.
What changes when you go live
A prototype has to run. A production system has to do four things a prototype never did.
It has to keep working. The model that scored well in your evaluation is not frozen. The world drifts: customers behave differently, the input distribution shifts, a vendor swaps the model under your API. An accuracy figure measured once is a photograph, not a guarantee. The question stops being does it work? and becomes is it still working, and how would I know if it stopped?
It has to fail visibly. A prototype fails on your laptop, where you see it. A production system fails in front of customers, quietly, and the worst failure is the one nobody notices until a complaint arrives. You need telemetry that surfaces the failure, not a dashboard of green ticks that hides it. Chapter 3’s lesson scales up: the dangerous output is the fluent one nobody checked, and the dangerous system is the one whose failures are invisible to the team that owns it.
Someone has to own it. A prototype is yours. A production system needs an owner, an on-call, and a named human who signs off on the red-quadrant outputs. “The model does this” is not an operating model; it is the shadow system from Chapter 3, now running in production. Name the owner before you ship, or the ownership defaults to nobody.
The legal and security questions become gates. Privacy and security obligations can apply during prototyping, before any production launch. Review the specific scope with the responsible specialists. Production readiness also needs documented controls: data-processing agreements in place, cross-border transfers lawful for the regions you serve, least-privilege tooling on any agent, and a human approval step on every action that is hard to reverse. Chapter 6 chose where the prompt may travel; production makes that choice auditable and enforced.
A production checklist (for a no-code practitioner)
This is what to demand of any system you move from proved prototype to live, phrased for someone who will hand it to engineers rather than build the engineering themselves.
- Watch the metric you evaluated on, sliced the way you evaluated it. Not aggregate accuracy. The per-class, per-segment, per-condition view is what caught the fault class collapsing under dim light (Chapter 9). In production that view is your canary; the aggregate is the lie that hid the failure in the first place.
- Re-evaluate on a schedule and on every change. A model swap, a new data source, a prompt rewrite: each is a reason to re-run the evaluation and re-place the output on the grid. Appendix A’s rule holds: re-place on evidence, on a schedule, never on hope.
- Have a kill switch, and know who can pull it. If the system cannot be turned off fast, it is not production-ready. Name the person, the channel, and the threshold at which they act.
- Make the human-in-the-loop a design, not a promise. A checkpoint that survives Monday-morning volume and deadline pressure is the only one that counts. The Delegation Trap (Chapter 3) does not disappear at launch; it intensifies, because production volume is exactly the pressure that turns a reviewer into a rubber stamp. Design the checkpoint so that rubber-stamping is hard, not easy.
- Log for audit, not for curiosity. Keep the records that let you answer, after a complaint, what did the system do, why, and who decided to trust it. The conversation log (Appendix B) is the prototype version of this discipline; production makes it a requirement.
- Sign off at go-live, on paper. The deployment decision (Chapter 6), the privacy architecture (Chapter 5), the security baseline (Chapters 7 and 11): each needs a named sign-off before launch, not a nod. A launch memo that records what was checked, by whom, and what the known limits are, is the artefact that separates a responsible deployment from a hopeful one.
Worked example: the router classifier at the gate
The constructed evaluation in Chapter 9 fails the fault-detection criterion in both lighting conditions. It does not earn a move into customer-facing routing. Tessera should revise and retest, or propose a separately approved shadow study in which predictions cannot close, route, or delay a real ticket.
In that shadow study, a reviewer checks every suggestion against the normal support process. Checking only suggestions labelled fault would miss the very failure the evaluation found: actual faults labelled healthy or degraded. Track the error counts and the time needed for review. If the review cannot keep up, stop the study rather than letting unchecked suggestions become actions.
A later candidate that meets the agreed criteria still needs the next gate’s evidence: representative data, a workable checkpoint, controlled access, monitoring, and a rehearsed fallback. Passing a small teaching test is not a reason to assume those controls exist.
Where this book stops
This book teaches you to specify a candidate, examine evidence and failures, and defend a bounded decision about its use. It stops short of teaching you to run it in production, because that is a different discipline.
What it does give you is the ability to see the line, to hand the system over with its limits honestly named, and to specify the oversight that must travel with it.