GuideBy the PhishSheriff team · December 2024 · 8 min read

AI-Generated Phishing: What It Means for Your Awareness Programme

Large language models have lowered the cost and raised the quality of phishing content. This guide explains how AI-generated attacks differ from traditional templates and how your simulation and awareness programme should respond.

Key points

How AI personalises targeted phishing at scale
Why easy simulations overstate your resilience
Using the attacks you actually receive as scenario material
Using AI for defence as well as attack

On this page

The economics have changed

For most of phishing’s history, attackers faced a trade-off: targeted spear-phishing worked but took hours of research and writing per target, while mass campaigns scaled but read like mass campaigns. Large language models remove much of that trade-off. An attacker can combine a public profile, a company news page and a supplier list and receive a fluent, relevant lure in moments, then repeat the process thousands of times with different output each time.

This affects defences that rely on repetition and style. Filters that look for identical messages sent in bulk see thousands of unique ones, and the language cues people were taught to spot are often absent. The baseline quality of phishing has risen.

What AI personalisation looks like

It helps to be concrete. A modern pipeline gathers public information about each target, such as role, manager, recent posts, projects mentioned in conference biographies and technologies named in job adverts, and uses it to shape the message. The result can mention a real expense platform, follow the phrasing of your industry, use the right language and arrive at a plausible moment: a fake supplier invoice at month-end, or a fake policy update in the week a real change was announced.

Language deserves particular attention. Awkward translation used to give away many lures aimed at non-English speakers. Models can now write fluent text in many languages, including Hindi, German, Japanese and Portuguese. If your organisation works in several languages but your simulations run only in English, part of your workforce is being prepared for a threat that has moved on.

The realism gap in older simulation programmes

If your simulations are easier than real attacks, your click rate understates your true risk, and everyone who relies on that number is misled. Template libraries built around fake parcel notices and prize draws reflect the commodity phishing of several years ago. Employees who reliably pass them may be recognising old templates rather than showing resilience to current attacks.

When a programme raises the difficulty of its exercises, the measured click rate often rises before it falls. That does not mean people got worse; the measurement became more realistic. Tell leadership in advance that this is a recalibration, and track progress from the new baseline.

Train against the attacks you actually receive

The most relevant scenarios are often the ones already aimed at you. Suspicious messages reported by employees show which brands, formats and pretexts attackers are using against your organisation and its suppliers. Used as scenario material, with every live link, attachment and tracking element removed, they help people rehearse against the threats they are most likely to meet.

Supplier impersonation is especially valuable to rehearse. When a lookalike of a key vendor’s domain appears, or a partner reports a compromise, the same pattern can inform exercises for the teams who deal with that supplier. Generic training teaches that phishing exists; scenario-specific training shows what it looks like when it comes for you.

Using AI for defence

The capability that helps attackers is also available to defenders. AI can generate simulations that reflect a person’s role and past results, so someone who previously fell for an invoice lure meets a harder invoice scenario rather than a random template. Used this way, personalisation strengthens preparation instead of enabling compromise. For examples to use in training, see spear phishing explained and five AI-enabled threats employees should recognise.

AI can also help after the exercise: short learning about the specific lure someone missed, delivered while it is still memorable, and faster triage of reported messages so analysts spend their time on conclusions rather than raw data. Programmes that use AI only to understand the threat, and not to adapt their own defences, are working with half the tools available.