AI-generated phishing attacks: How AI changes social engineering

Large language models have eliminated the grammatical errors and generic templates that defenders relied on to catch phishing. AI-generated phishing is personalized, contextual, and produced at scale. What that means for detection and how security teams should adapt.

What is AI-generated phishing?

AI-generated phishing uses large language models to create social engineering content (emails, messages, fake login pages, voice scripts) that is personalized to individual targets and produced at volume. Instead of reusing a template with obvious spelling mistakes across thousands of recipients, an attacker can generate unique, contextually relevant messages for each target based on their role, company, recent activity, and communication style.

The Verizon 2025 Data Breach Investigations Report identified phishing as a component in 36 percent of breaches, and researchers noted a measurable increase in the sophistication of phishing content consistent with LLM-generated language. The IBM X-Force 2025 Threat Intelligence Index documented a 45 percent increase in phishing campaigns using AI-generated content compared to the prior year. These are not fringe attacks; they are becoming the standard approach for well-resourced threat actors.

The operational problem is straightforward: the signals that traditionally identified phishing (grammatical errors, generic greetings, implausible scenarios, formatting inconsistencies) are disappearing. AI-generated phishing reads exactly like legitimate business communication, because the same LLMs that write professional emails are now writing phishing emails. Detection must shift from content inspection to behavioral analysis of what happens after a user interacts with the message.

How AI phishing attacks work

The workflow behind an AI-powered phishing campaign is more systematic and scalable than traditional phishing. Each stage has detection opportunities.

Stage 1: Automated reconnaissance

The attacker uses AI tools to scrape and synthesize publicly available information about targets. LinkedIn profiles provide job titles, reporting structures, and recent career changes. Company websites reveal organizational structure, recent announcements, and technology stack. Social media posts reveal communication patterns, interests, and personal details. AI processes all of this into a target profile in seconds. Work that would take a human researcher hours per target.

Stage 2: Personalized content generation

Using the target profile, the attacker prompts an LLM to generate a phishing email that matches the target's context. A finance team member might receive an email about a vendor payment issue, referencing the company's actual ERP system. An IT administrator gets a message about an expired SSL certificate, written in the technical tone they would expect from a colleague. The content is unique per target; there's no shared template for a content-based filter to pattern-match against.

The scale advantage is clear here. A human attacker crafting spear-phishing emails might produce 10 to 20 quality messages per day. An LLM can generate thousands of personalized variants in minutes, each adapted to a different target's role, company, and likely concerns. The economics of spear-phishing have changed fundamentally. What was once an expensive, targeted attack is now available at commodity phishing volumes.

Stage 3: Multi-channel delivery

AI-generated social engineering is not limited to email. Attackers use the same models to generate convincing Teams messages, Slack DMs, SMS texts, and even deepfake voice calls. The CrowdStrike 2025 Global Threat Report documented cases where attackers used AI voice cloning to impersonate IT help desk staff in vishing (voice phishing) attacks, calling employees to request MFA approval for attacker-initiated login attempts. Each channel has different detection tools and visibility, and attackers exploit the gaps between them.

Stage 4: Credential harvesting and initial access

The phishing message directs the target to a credential harvesting page, a malicious attachment, or a social engineering pretext for sharing credentials or approving MFA. This is where detection shifts from the email layer to the identity and access layer - Active Directory logs, authentication events, cloud access logs, and endpoint telemetry become the primary detection surfaces.

Why traditional phishing detection fails against AI

Email security gateways and traditional anti-phishing tools were designed for a different era of phishing. Their specific failure points against AI-generated content explain why detection needs to shift.

Content-based signatures are irrelevant

Traditional filters match known malicious phrases, sender reputation scores, and template fingerprints. AI-generated content is unique per message; there's no phrase to match and no template to fingerprint. The sender infrastructure may be compromised legitimate accounts rather than blocklisted domains. Every signal that content-based detection relies on is absent.

URL and domain analysis has a window

Link analysis catches known malicious URLs but struggles with freshly registered domains, compromised legitimate sites, and URL shorteners. Attackers using AI can generate thousands of unique landing pages as easily as they generate unique email content. Domain reputation systems have a lag. New domains have no reputation, which is different from having a bad reputation, and most email gateways default to allowing unknown domains rather than blocking them.

Header analysis catches infrastructure, not content

SPF, DKIM, and DMARC verification catches domain spoofing, but many AI phishing campaigns use compromised legitimate email accounts or freshly provisioned accounts on real email services. The headers pass all authentication checks because the sending infrastructure is technically legitimate, but the content and intent are malicious, and header analysis cannot assess intent.

AI phishing simulators: Testing your defenses

The same AI capabilities that attackers use to generate phishing are available to security teams for defensive testing. AI phishing simulators generate realistic campaigns that test employee resilience against the exact attack patterns organizations face in production.

What legitimate AI simulators do

Modern phishing simulation platforms use LLMs to generate personalized test campaigns that mirror real attacker methodology. They scrape public information about employees (with organizational consent), generate role-appropriate pretext scenarios, and deliver test emails that are indistinguishable from real AI-powered phishing attempts. The goal is to measure how employees respond when the phishing content is too good for traditional "spot the error" training to work.

Metrics that matter

• Click-through rate by department: Which teams interact with phishing at higher rates? This identifies targeted training priorities.
• Credential submission rate: Clicking a link is one thing; entering credentials on a fake login page is the actual breach point. This metric matters more than click rates.
• Time to report: How quickly do employees flag suspicious messages to the security team? Organizations with mature security cultures see 20 to 40 percent report rates; those without see single digits.
• Repeat offender identification: Users who fail multiple simulations need additional support (not punishment), and their accounts may warrant enhanced monitoring in UEBA risk scoring.

Connecting simulation results to SIEM monitoring

The real value of AI phishing simulations is not the training report; it is the data that feeds back into your SIEM detection strategy. If simulations show that finance team members click at 3x the organizational average, that information should inform monitoring priority. Log360 allows you to configure risk weighting in the UEBA engine based on role-based risk assessments, giving higher-risk user groups tighter anomaly detection thresholds without manually babysitting their accounts.

Detecting AI phishing through behavioral signals

When the phishing email itself is undetectable by content analysis, detection shifts to what happens after the user interacts with it. Post-delivery behavioral signals are the most reliable indicators of successful phishing, and they work no matter how polished the phishing content was.

Authentication anomalies

After successful credential harvesting, the attacker uses the stolen credentials from their own infrastructure. This creates authentication events that UEBA behavioral analytics can detect: logins from new geographic locations, impossible travel (logins from two distant locations in a timeframe that makes physical travel impossible), first-time device registrations, and authentication from IP ranges inconsistent with the user's history. These signals appear in Active Directory logs, cloud authentication logs, and VPN connection records.

Post-compromise behavior patterns

A compromised account behaves differently from a legitimate user. Within the first hours after credential compromise, attackers typically perform email inbox searches (looking for sensitive data and email threads to hijack), Active Directory enumeration (mapping the organization), access to file shares or cloud storage the user does not normally touch, and lateral movement attempts using discovered credentials or tokens. Correlation rules that chain these signals ("new location login" + "inbox rule creation" + "AD enumeration" within a 2-hour window) produce high-confidence alerts for credential compromise whether the delivery was email, SMS, or voice.

Email account manipulation

Attackers who compromise email accounts through phishing frequently create inbox rules to hide their activity: auto-forwarding emails, moving messages from security teams to the trash, or hiding replies from targets in business email compromise schemes. Monitoring Microsoft 365 audit logs for inbox rule creation, mail forwarding changes, and delegate access modifications provides early detection of email account takeover following successful phishing.

Building a detection strategy for AI-era phishing

Content-based detection is not dead; it still catches commodity phishing. But for AI-generated campaigns, your primary detection layer must be behavioral. A layered approach covers both.

• Keep email gateway defenses in place: SPF, DKIM, DMARC, URL sandboxing, and attachment analysis still catch a significant percentage of phishing (the campaigns that have not yet adopted AI). Do not remove these layers; add behavioral detection on top of them.
• Deploy UEBA with authentication focus: User and entity behavior analytics should monitor authentication patterns across all identity sources - Active Directory, Azure AD, cloud applications, VPN. Behavioral baselines detect credential compromise however the credentials were stolen.
• Build correlation rules for phishing chains: Configure SIEM correlation rules that chain phishing-related signals: "email with link clicked" + "credential submission on external site" + "new location login within 4 hours" + "inbox rule creation." Individual signals are weak; the chain produces high-confidence alerts.
• Monitor identity-layer changes: Microsoft 365 audit logging, AD change monitoring, and cloud IAM logs reveal the account manipulation that follows successful phishing: inbox forwarding rules, delegate access, new MFA device registrations, and OAuth application consent grants.
• Run AI phishing simulations quarterly: Use AI-powered simulators to test your organization's resilience against realistic phishing. Feed results into your SIEM strategy - adjust UEBA risk weights, refine correlation rules, and prioritize monitoring for high-risk user groups.
• Integrate AI investigation for triage speed: When phishing alerts fire, AI-guided investigation can analyze the affected user's behavior across all log sources, mapping the post-compromise timeline from initial access through lateral movement. Manual investigation of the same scope takes 20 to 40 minutes per alert.

The defensive AI advantage

AI is available to both sides. While attackers use LLMs to generate phishing, defenders can use AI to detect and investigate the results faster than manual processes allow.

AI-guided investigation after phishing alerts

When a suspected phishing compromise triggers an alert, Log360's Zia Insights performs entity-deep analysis of the affected user. The AI correlates the user's authentication events, endpoint activity, email behavior, cloud access patterns, and file operations across the entire log corpus. It produces a structured investigation narrative: when the compromise likely occurred, what the attacker accessed, which MITRE ATT&CK techniques are present, and what containment steps are recommended. The analyst reviews and decides. The AI does not autonomously contain the account. This approach compresses a 30-minute manual investigation into a 3-minute review.

UEBA risk scoring as a phishing probability multiplier

Log360's UEBA engine assigns risk scores to users based on behavioral deviations from their baseline. When a user who normally logs in from one city suddenly authenticates from another country, accesses applications they have never used, and creates an inbox forwarding rule, all within a 2-hour window, the composite risk score spikes. That risk score, combined with a recent phishing alert for the same user, gives the analyst a high-confidence triage signal without wading through raw logs.

Frequently asked questions