7 Prove? Evidence Collection and Testing
You can read your way to a suspicion. You can only evidence your way to an opinion.
🗂️ Tessera Case: Operating records arrive
Act I identified the revocation deadlines and the evidence needed to test them. E04 now supplies 48 leavers and their disable events. E05 adds incident and cause records. These are documentary operating evidence, even when read from a desk. Your task is to establish what each record supports and retain the reasoning so another reader can reproduce it.
7.1 The act break: from suspicion to proof
The staged release of records changes what you can conclude. It does not remove uncertainty. A timestamp may refer to the wrong event; a join may omit service credentials; a complete-looking export may cover only one business unit.
Access is a prerequisite for some procedures, not assurance in itself. Start with the objective and examine whether the available evidence can answer it. Missing evidence belongs in the result, rather than being supplied through inference.
7.2 Four kinds of evidence
Observational evidence records what you saw: a revoked badge denied at a reader, or a tested access decision. It can be strong for that event. It may be less representative if staff alter their behaviour while observed.
Documentary evidence includes policies, logs, signed checklists, contracts and external reports. A policy supports a design claim; historical logs may support operation over a period. Reliability depends on creation, access, retention and alteration controls. “System generated” is not enough if the system can omit events or the administrator can rewrite its output.
Analytical evidence is work you perform on inputs: reconciliation, elapsed-time calculations, trend analysis or re-performance of a review. It can expose an inconsistency, but inherits errors in those inputs. Producing a spreadsheet yourself does not make the source complete.
Oral evidence includes interviews and explanations. It helps establish context and locate records. Memory, incentives and ambiguity can limit it. Corroborate consequential claims, especially when an objective record should exist.
There is no universal ranking that decides every case. Independent, contemporaneous and well-controlled records often deserve more weight, but relevance comes first. A current third-party report on the wrong service cannot answer a question about Tessera’s leavers. A reliable internal event log can.
7.3 Sufficiency and appropriateness
Sufficiency concerns how much evidence is needed for the conclusion. Appropriateness concerns its relevance and reliability. Evaluate both against the assertion, risk, period and intended reliance.
One event may settle a narrow claim about that event. It cannot establish that the same control operated for every event in a year. Conversely, a large collection of irrelevant screenshots adds little assurance. State the population and what the evidence actually covers.
A screenshot without account, time or provenance may still be a lead. It is weak support for a deployment-specific claim. A summary can help navigate raw records; preserve and inspect the underlying records before relying on details that summarisation might omit.
7.4 Sampling: why you cannot test everything, and how to size what you do
Define the population before selecting items. For the fixed case, it is all 48 departures from 1 June 2025 to 31 May 2026, including their separately managed keys. The pack stipulates reconciliation with payroll for the exercise. In real work, obtain the independent population, inspect exclusions and reconcile identifiers yourself.
For a large population, attribute sampling tests whether each selected item meets a defined condition. The sampling design must say how items are selected, what deviation means, how much sampling risk is accepted and how exceptions affect the conclusion. A convenience sample does not support a statistical confidence claim.
7.4.1 A numerical example
Consider a separate, large population of similar independent events. Suppose you expect no deviations, want a zero-deviation acceptance design and choose a tolerable rate of 5%, with a one-sided sampling risk of 5%. If the actual deviation probability were 0.05, the chance of observing zero deviations in \(n\) independent trials would be \(0.95^n\).
Choose the smallest \(n\) for which:
\[ 0.95^n \leq 0.05, \qquad n \geq \frac{\log(0.05)}{\log(0.95)} = 58.404\ldots \]
Round up to 59. With zero observed deviations, the one-sided exact binomial upper bound is approximately 4.95%. This is not a claim that the control is 95% reliable or that the probability of a breach is 5%.
The calculation assumes a binomial model and suitably independent representative selection. A finite population sampled without replacement requires an appropriate finite-population method; clustered events and different risks may require stratification or another design. If deviations are expected or found, this zero-acceptance example is not a complete evaluation plan. Agree the method and decision rules before drawing a sample.
7.4.2 Why Tessera uses a census
There are only 48 leavers, and the key-coverage question is important enough to inspect each. We test 48 of 48: a census, not a 59-item sample compressed into a smaller population. A census removes selection-based sampling uncertainty for those recorded items. It does not remove missing-population, recording, procedure or interpretation error, and it does not predict future operation.
7.5 The five testing procedures
Inquiry: ask how a record was created and where exceptions are handled. Record what was claimed and what still needs corroboration.
Observation: watch an authorised control execution. Record the environment, participants and conditions; do not imply that the observed event represents an untested year.
Inspection: examine the relevant records or assets. For revocation, establish whether an event means disablement, deletion, last use or merely a request. A key’s “last used” timestamp does not prove when the key was revoked.
Re-performance: independently repeat a control or check, within the approved scope. Reconcile the complete leaver list with credential ownership and disable events rather than relying solely on a signed checklist.
Re-computation: check arithmetic, such as elapsed time and exception rates. Keep timezone and boundary conventions explicit. A two-hour rule means elapsed time no greater than two hours here; a result exactly at two passes.
These procedures answer different questions. Re-performing a calculation with incomplete inputs is not stronger than a reliable source record merely because it appears later in this list. Combine methods to address the assertion and its principal failure modes.
An illustrative draft proposes “test 25 leavers, verify manager notification, and confirm that old accounts are disabled today.”
The sample has no stated design. E04 is small enough for a census. Manager notification has no supplied event record and is not the revocation criterion. A current-state check cannot establish a past disable time. Replace the steps with an ID reconciliation, inspection of both credential families and elapsed-time calculations against both historical policies.
The correction is checkable from the pack; no forecast about how every model behaves is needed.
7.6 Chain of custody
Record who obtained each item, when, from which system and by what query or export. Retain timezone, filters, permissions and transformations. Preserve the original and work on a controlled copy. Log transfers and restrict access according to the engagement’s retention policy.
A SHA-256 digest can detect a change relative to the recorded digest. It cannot tell you whether the original extract was truthful, complete or collected by the claimed person. Protect the manifest as well as the files. A digest invented for an illustration is not a reproducible checksum.
The book’s source provides full checksums for its electronic teaching records. Those are packaging-integrity checks, not forensic provenance. For E04, use the manifest’s synthetic-source description and capture date; do not record the author as a real-world auditor who extracted an actual client log.
7.6.1 One reproducible working paper
For L007, termination is 9 January at 16:00 AWST. Identity access was disabled at 17:00 that day. The separate key was disabled on 25 February at 16:00: 47 days = 1,128 hours later. The latest disable event determines full revocation. It exceeds both P-IT’s 24-hour and P-HR’s two-hour historical deadlines.
For L044, the key delay is six hours. It passes P-IT and fails P-HR. That is why silently choosing the less demanding policy changes the result.
For the full population, three records exceed 24 hours: L007, L019 and L031. Four exceed two hours: those three plus L044. Record 3/48 = 6.25% and 4/48 ≈ 8.33%, each with its criterion. These are observed deviations in the supplied historical population, not probabilities of misuse or a future breach.
7.7 Auditing AI: can you trust a log an AI produced?
Distinguish a deterministic event recording that a model was called from the model’s generated account of what happened. A generated summary may omit, misstate or infer details. Compare it with the original event, the action actually taken and any downstream state change.
To evaluate an AI-assisted alert decision, define the desired decision, relevant classes and an independently reviewed reference set. Preserve uncertain labels and disagreements. Measure missed incidents as well as false alarms; a high overall accuracy can conceal failure on rare serious cases. Record model/version, configuration, period and operating context.
The fixed pack does not supply an alert-evaluation dataset, so it supports no conclusion about Tessera’s anomaly detector. Treat this as a method extension. Do not add a successful AI-control test to the final opinion unless the underlying inputs and results exist.
7.8 Interviewing for evidence
Prepare from the objective. Ask an open process question, then request an example and its record. “Walk me through the last contractor departure” is more useful than a question that supplies the desired answer.
Keep three columns: claim, record requested, corroborated result. Differences between accounts can reflect scope, dates, memory or process variation. Investigate them rather than deciding in advance which person is evasive.
The interview guide provides a paper dialogue and optional live-site route. A difficult conversation is not proof of a defective control. A fluent answer is not proof of a sound one.
Time: 35 minutes. No AI or code required. Test all 48 E04 rows. Retain the population definition, policy-specific result counts, four exceptional IDs, a full L007 calculation and a provenance note. Separately reproduce the 59-item sampling example with a calculator, or explain its assumptions from the worked calculation. Label the actual case test a census.
7.9 The question, answered
A supported result states what was tested, what the records show and what remains uncertain. The next chapter uses these results to assess policy compliance and review coverage.
7.10 Take it to the AI
Optional, after the paper test:
Role: Review a trainee’s audit calculation.
Task: Check whether a proposed sample and conclusion follow from the supplied facts.
Context: The complete fictional population contains 48 leavers. Three exceed 24 hours; four exceed two hours. All 48 were tested. A draft says “a statistically derived sample proves a 6% chance of a future breach.”
Format: Identify each unsupported claim, correct the arithmetic and give a bounded two-sentence conclusion. Use no additional case facts.
Check that the response separates census, historical deviation and future incident risk. A correct calculation still needs the source records behind it.