In early 2024, a finance employee received an email from what appeared to be the company CFO. They were not sure, so they joined a video call — the CFO was there, senior colleagues too. Everything looked right. Everyone sounded right. They approved a $25 million transfer.
None of those people were real. Every face, every voice — AI generated. The email that started it? Perfectly written. Nothing to catch.
If your security awareness training is still telling people to look for typos, you have a problem.

First, Let Me Show You What Changed
Here is a phishing email from 2018:
“Dear Customer, You’re account has been suspended!! Click here IMMEDIATELY to verify or you will loose access. Regards, Support Team”
Here is one written by an AI model today, targeting a specific employee at a specific company:
“Hi Sarah, following up on the vendor onboarding discussion from last Tuesday’s call — the procurement team needs the updated bank details for Meridian Consulting before the Thursday cut-off. Can you action this today? I’m in back-to-back meetings but James from finance can take it from here if needed. Thanks, David”
Same goal. Completely different game.
The second email has no malicious link yet. It is just building trust, waiting for a reply, starting a conversation. The attacker knows Sarah’s name, knows there was a vendor onboarding discussion, knows who David and James are. All of that came from LinkedIn, a press release, and a data breach.
That research used to take hours. AI does it in minutes
How Attackers Are Actually Using AI — The Real Workflow
Let me walk you through what an attacker’s workflow actually looks like today. This is not theoretical.
Step 1 — Reconnaissance on Autopilot
Before writing a single word, attackers feed targets into AI tools that scrape:
- LinkedIn profiles — job titles, recent posts, who reports to who
- Company press releases — new hires, software migrations, partnerships
- Data breach dumps — old passwords, email formats, internal details
- GitHub repositories — employee names in commits, internal tool names
An AI model then stitches this into a profile: “Target is Sarah Chen, Accounts Payable Manager at Meridian Corp. Recently posted about implementing a new ERP system. Reports to CFO David Park. Company uses Microsoft 365.”
That profile feeds directly into the email.
Step 2 — Email Generation at Scale
Tools like FraudGPT and WormGPT — dark web LLMs with no ethical guardrails — take that profile and generate hundreds of personalized phishing emails in one go. Each one unique. Each one tailored. Each one grammatically flawless.
IBM ran an experiment in 2025 that tells you everything. Their security team challenged human social engineers and an AI to create a phishing campaign. The humans took 16 hours. The AI needed 5 prompts and 5 minutes — and the click rates were comparable.
Step 3 — The Email Hits the Inbox
Because the email is perfectly written and contains no malicious links or attachments at this stage, it sails straight through traditional email filters. Those filters were built to catch spelling mistakes, known malicious domains, and suspicious attachments. A well-written email with no links fails exactly zero of those checks.
The Numbers Are Hard to Ignore
I know statistics can feel abstract, so let me put these in context:
- 83% of phishing emails are now AI-generated — KnowBe4, 2025
- 54% click-through rate on AI phishing vs 12% for traditional campaigns — Vectra AI
- $215.8 million in direct phishing losses in 2025, up from $70 million in 2024 — FBI IC3
- AI-generated emails achieve a 30–44% click rate vs 19–28% for human-written — Harvard / Black Hat research
- Phishing attacks increased 4,150% since ChatGPT launched
The last one is the one that should make you pause. 4,150%. That is not a trend. That is a step change.
The Dark Web Toolbox — What Attackers Are Actually Using
You do not need to build anything yourself if you want to run an AI phishing campaign today. Here is what is available:
| Tool | Type | What It Does |
|---|---|---|
| WormGPT | Dark web LLM | GPT-based model fine-tuned on malware and phishing data. No refusals, no guardrails |
| FraudGPT | Dark web LLM | Marketed specifically for BEC and phishing. Subscription-based on Telegram |
| GhostGPT | Dark web LLM | Newer variant, cheaper, widely available. Generates phishing templates, malware code |
| EvilGPT | Dark web LLM | Focuses on social engineering scripts and vishing (voice phishing) support |
These are not obscure tools. FraudGPT had over 3,000 confirmed subscribers within its first month. They are marketed, supported, and updated like legitimate SaaS products.
Open Source Tools Defenders Use to Understand This — and Test Their Own Defenses
These tools are for defenders — security teams who want to understand how AI phishing works and test whether their own users and systems would catch it. All of them are widely used in red team exercises and security awareness programs.
| Tool | What It Is | Why It Matters | GitHub |
|---|---|---|---|
| GoPhish | The most widely used open source phishing simulation framework. Free, self-hosted, actively maintained | Lets you run controlled phishing campaigns against your own organization. Pair it with AI-generated content to test if your users would fall for modern well-written lures — not just obvious templates | gophish/gophish |
| SpiderFoot | Open source OSINT automation tool. Aggregates data from hundreds of sources — LinkedIn, breach databases, DNS records, social media | This is exactly what attackers automate before writing phishing emails. Run it against your own org and see what an attacker sees before they write a single word | smicallef/spiderfoot |
| TheHarvester | Classic OSINT tool focused on email addresses, employee names, subdomains, and open ports from public sources | Attackers use this to build their target list before feeding it into an AI. Knowing what is publicly exposed is step one in reducing your phishing attack surface | laramies/theHarvester |
| Ollama | Run open source LLMs locally — Mistral, LLaMA, Phi — entirely offline on your own machine | Security researchers use this to study how LLMs generate phishing content without sending data to external APIs. Test it safely, learn from it locally | ollama/ollama |
| Mailoney | A simple SMTP honeypot that captures inbound phishing attempts | Deploy it and watch what lands. Real AI phishing samples hit your honeypot and you can study them directly — grammar patterns, lure themes, sending infrastructure | phin3has/mailoney |
What Does AI Phishing Actually Look Like — Side by Side
| Feature | Old Phishing | AI Phishing |
|---|---|---|
| Grammar | Spelling mistakes, awkward phrasing | Flawless, natural writing |
| Personalization | Generic — “Dear Customer” | Specific — uses real names, roles, recent events |
| Context | None — random lure | References real internal details scraped from public sources |
| Volume | Limited by attacker’s time | Thousands of unique emails per hour |
| Links / Attachments | Always present | Often absent in early messages — builds trust first |
| Tone | Urgent and alarming | Calm, professional, conversational |
| Detection by filter | Caught regularly | Passes most traditional filters |
How to Actually Detect It
Since you cannot rely on grammar and spelling anymore, here is where the detection surface shifts:
1. Look at behavior, not content AI can write a perfect email. It cannot fake that the sender’s domain was registered 48 hours ago, or that the sending IP has never emailed your organization before, or that the reply-to address does not match the from address.
2. Enforce DMARC at p=reject This does not care how good the email looks. It checks whether the sending infrastructure is actually authorized to send on behalf of that domain. AI cannot fix an authentication failure.
3. Watch for unusual urgency patterns AI phishing still needs to drive action. It still needs a hook. Watch for requests that bypass normal process — wire transfers, credential sharing, installing software — even when the email itself looks completely normal.
4. Out-of-band verification for anything financial If an email asks for money movement, call the person. On a number you already have. Not one in the email.
5. Train users on the new red flags Not typos. Not bad logos. The new red flags are: unexpected requests that bypass normal process, emails referencing recent internal events they should not know about, and any urgency around financial or credential actions.
The Uncomfortable Truth
Traditional security awareness training taught people to look for bad writing. Attackers no longer write badly.
The $25 million Arup attack worked not because the employee was careless. It worked because everything looked exactly right — the faces, the voices, the email, the context. No amount of “check for spelling mistakes” training would have helped.
The training has to change. The detection has to change. And the assumption that a well-written email is a trustworthy one has to go.
AI did not just make phishing easier. It made the old defenses irrelevant.
⚠️ Disclaimer: This article is intended for educational and defensive security purposes only. All tools and techniques mentioned are for defenders testing their own environments — not for malicious use



































