11 Human Factors & Emerging Trends

11.1 What this chapter covers

By the end you should be able to:

Explain how human decisions interact with system design, and why blame alone does not repair a weakness.
Describe social engineering and phishing, and the defences that actually work.
Reason about how non-technical people make security decisions with the information they’re given.
Discuss emerging trends, including AI’s double role as both a tool for attackers and a tool for defenders.

11.2 The weakest link, and why blaming the user fails

Calling people the weakest link treats a complex system as a single chain. More usefully, ask what the interface, permissions and work process made likely. The claim that “the users are the weakest link” is a trap: a framing that excuses bad security. When a well-crafted phishing email fools an employee, the instinct is to blame the employee: they should have known better. But look closer. They were doing their job, quickly, inside a system that made the wrong action easy, the right action unclear, and the difference between the two nearly invisible. Blaming the person lets the system off the hook. A team that blames users stops fixing the thing they could fix.

A more useful stance examines the system. People will make mistakes. That is a constant, and good security is designed around it rather than trying to train it away. This is assume breach applied to human beings: assume that, sooner or later, someone will be fooled, and build so the organisation survives it. That single shift, from “make users perfect” to “make mistakes survivable”, is the difference between security that works and security that just apportions blame after the fact.

11.3 Social engineering and phishing

Social engineering is hacking the human instead of the machine: manipulating a person into doing what the attacker wants. It works because it targets the mental shortcuts we all rely on to get through the day. An attacker invokes authority (“this is IT, we need your password”), urgency (“the account will be closed in an hour”), fear, trust, or plain helpfulness. The victim, trying to be efficient and cooperative, complies. They are usually behaving well by the ordinary rules of work; the attacker has pointed those good instincts in the wrong direction.

Phishing is social engineering at scale: the deceptive email or message, cast wide, that lures people into revealing credentials or running malware. Spear-phishing is the targeted, personalised version aimed at a specific person, and it is more convincing. These attacks succeed because they exploit how humans are built to work. Their victims are not foolish.

The defences that help are instructive, because the obvious one is the weakest. Awareness training matters, but you cannot train people to be perfect forever, and a defence that requires every human to be vigilant every time will eventually fail. The stronger approach is to reduce the reliance on human vigilance: use technical controls that make a mistake less catastrophic. Multi-factor authentication means a phished password alone isn’t enough, which is the point of the authentication chapter. Email filtering can remove some lures before they reach a person. Firm verification procedures for high-risk actions mean no single email can move money or hand over data without an independent check. Best of all is to make the safe path the easy path, so that doing the secure thing is also the convenient thing. The goal is a system in which an ordinary human error does not become a disaster, rather than a workforce that never errs.

11.4 Communicating risk to people who aren’t technical

A quieter human-factors failure is that we push security decisions onto people who have no way to make them well. The classic example is the browser certificate warning. When your browser says a site’s certificate can’t be trusted, it is telling you something important. It could not verify you’re talking to who you think you are, using the identity-proving cryptography from the cryptography chapter. That warning might be the only thing standing between you and a man-in-the-middle attacker.

Some users bypass warnings when the language is unclear, the warning interrupts urgent work, or past experience has taught them to discount it. Behaviour varies with the warning and context; do not assume most users always ignore it. This is a design failure, not the user’s. A consequential security decision has been handed to someone without the information or context to make it, in words that mean nothing to them. Good security gives people clear information and, wherever possible, doesn’t force impossible judgements on them.

11.5 Emerging trends: AI on both sides

AI can assist both attackers and defenders, but capability claims need a date, a task and evidence. It can lower the effort of drafting messages, summarising data or producing code. That does not make every message flawless, every exploit effective or an operation limitless.

For attackers, plausible personalised language and synthetic audio can make familiar verification shortcuts less reliable. For defenders, models may help triage reports and interpret logs, but false alarms, missed attacks and untrusted inputs still need evaluation. Logs and messages can contain hostile instructions as well as sensitive information, so an assistant must not be given authority merely because it produced a confident answer.

The NCSC assessment of AI and cyber threats to 2027 describes expected changes in attacker capability. Treat it as a dated assessment with uncertainty, not proof that every forecast has occurred.

At Harbour, an apparent manager’s voice is not enough to approve an account reset or bulk export. Use a known contact route and the agreed approval process. This remains useful whether the message was written by a person or generated by a model. Give staff a reporting path that does not punish them for raising a concern that turns out to be harmless.

11.6 Where this connects

This chapter connects back across everything before it. Human decisions influence the design, operation and recovery of technical controls, so human factors belongs throughout the book. Authentication is the clearest case: MFA is a technical control that makes a human mistake survivable. And that same thread, the human running through every stage, is part of how the next chapter ties the whole book into a single lifecycle.

11.7 Questions to consider

“Users are the weakest link.” Argue against this framing: whose fault is a successful phish, really?
What does a browser certificate warning tell you? Explain why someone might bypass it and propose a safer route the organisation could offer.
Give one way AI helps attackers and one way it helps defenders. Which side do you think it favours, and why?

11.8 Make reporting usable

In 10 minutes, write a response to this fictional message: “Mara here. Approve the login prompt now so I can finish the bookings.” Use a known contact route and the agreed approval process. Add a non-blaming report for someone who already approved it. Success means prompt escalation and independent verification, without using contact details supplied by the message.

Case inputs and checks: Harbour dossier.

11.9 Take it to the AI

Optional: attempt the paper task first. Use only the fictional case, never real messages, credentials, personal information or internal logs. Check any external claims against the primary sources; do not deploy generated rules.

Act as a reviewer of Harbour’s reporting procedure. Use only the fictional message and case facts. Ask me what someone who already approved a prompt should do next. Then identify one obstacle in my procedure and one change that makes reporting easier without bypassing verification.