AI-Powered Cyber Threats and Defensive AI: What Is Really Changing
AI is making existing attacks faster, cheaper and more convincing rather than creating entirely new ones. Government and industry reports describe attackers using generative AI to write phishing messages in any language, research targets, debug malicious code, create fake profiles and clone voices for fraud.
Topic: AI in Cybersecurity in Cybersecurity & Ethical Hacking · Explainer
Also covers: Cyber Threats, Scams & Online Safety
Quick answer: AI is making existing attacks faster, cheaper and more convincing rather than creating entirely new ones. Government and industry reports describe attackers using generative AI to write phishing messages in any language, research targets, debug malicious code, create fake profiles and clone voices for fraud. The UK's NCSC assessed in 2024 that AI will almost certainly increase the volume and impact of cyber attacks over the following two years, mainly by uplifting less-skilled actors. Defenders use AI to triage alerts, detect anomalies, summarise incidents and find vulnerabilities before attackers do. The fundamentals, MFA, patching, verification of payment requests and good logging, still stop most AI-assisted attacks.
Key takeaways
- Most observed attacker use of AI is productivity: writing, translating, researching, coding help.
- Fraud using cloned voices and video is the most visible new impact for individuals and businesses.
- Defensive AI helps with scale: too many alerts, logs and code to review manually.
- AI systems themselves are now targets, through prompt injection and data poisoning.
- Basic controls remain the best defence against AI-assisted attacks.
Table of Contents
- What Do Official Reports Say About Attackers' Use of AI?
- Which Attacks Are Changing Most?
- How Are AI Systems Themselves Attacked?
- How Is Defensive AI Used?
- How Do Attacker and Defender Uses Compare?
- What Should Organisations Do Now?
- Which Frameworks Help Manage AI Threats?
- How Do You Detect AI-Assisted Phishing in Practice?
- References
What Do Official Reports Say About Attackers' Use of AI?
- UK NCSC (January 2024): assessed that AI will almost certainly increase the volume and heighten the impact of cyber attacks, with the biggest uplift in reconnaissance and social engineering.
- Google Threat Intelligence Group (January 2025): reported that government-backed actors used Gemini mainly for research, troubleshooting code and content creation, and did not observe novel capabilities that changed the threat landscape.
- OpenAI threat reports: describe disrupting accounts that used its models for influence operations, scams and malware debugging.
- FBI IC3 (December 2024): warned that criminals use generative AI for fraud, including voice cloning and fake identity documents.
Which Attacks Are Changing Most?
| Attack | What AI changes | Defence |
|---|---|---|
| Phishing | Fluent, personalised messages in any language | Phishing-resistant MFA, reporting culture |
| Voice and video fraud | Cloned voices of executives or relatives | Call-back verification, payment controls |
| Reconnaissance | Faster research on targets and technologies | Limit public exposure, monitor brand abuse |
| Malware development | Help writing and debugging code | EDR, application control, patching |
| Vulnerability exploitation | Faster analysis of patches and public code | Patch quickly, especially internet-facing systems |
Read more on deepfake and AI-enhanced social engineering.
How Are AI Systems Themselves Attacked?
Organisations deploying AI create new attack surface: prompt injection that makes a chatbot leak data or misuse tools, poisoning of training data, theft of models and abuse of AI agents with broad permissions. MITRE ATLAS catalogues these techniques. See our explainer on prompt injection.
How Is Defensive AI Used?
- Detection: machine learning in EDR, email security and network tools has flagged unusual behaviour for years.
- Triage and summarisation: assistants summarise alerts and draft incident timelines for analysts.
- Vulnerability discovery: AI-assisted fuzzing and code analysis; DARPA's AI Cyber Challenge had teams build systems that find and patch bugs in open source software automatically.
- Threat intelligence: extracting indicators and techniques from long reports.
Analysts remain responsible for decisions; AI output should be checked like any other source.
How Do Attacker and Defender Uses Compare?
| Area | Attackers | Defenders |
|---|---|---|
| Scale | More phishing and scam variants | More alerts and logs reviewed |
| Skill | Lower-skilled actors gain capability | Junior analysts work faster with guidance |
| Code | Debugging malicious tools | Finding and fixing vulnerabilities |
| Limits | Model safeguards, detection by providers | Errors, false positives, data protection |
What Should Organisations Do Now?
- Deploy phishing-resistant MFA for email, VPN and admin access.
- Require call-back verification for payments and bank detail changes.
- Patch internet-facing systems quickly; exploitation windows are shrinking.
- Train staff with examples of AI-written phishing and cloned voices.
- Govern your own AI deployments: inventory, data rules, least privilege for agents.
- Evaluate defensive AI on your own data before relying on it.
Related: AI in DevSecOps and AIOps. WebAsha offers AI security training.
Which Frameworks Help Manage AI Threats?
| Framework | Use |
|---|---|
| NIST AI RMF | Govern and manage risks of AI you deploy |
| NIST AI 100-2 | Taxonomy of attacks on machine learning systems |
| NCSC and CISA secure AI development guidelines | Secure design, development, deployment and operation |
| OWASP LLM Top 10 | Application-level risks such as prompt injection |
| MITRE ATLAS | Adversary techniques against AI systems |
How Do You Detect AI-Assisted Phishing in Practice?
Because AI removes spelling mistakes, detection has to rely on signals AI does not change: sender authentication, new domains, unusual requests and behaviour after a click. A simple check of message headers for authentication results:
# Save a suspicious email as .eml, then look at authentication results
grep -iE "^(Authentication-Results|Received-SPF|DKIM-Signature|From|Reply-To):" suspicious.eml
A "Reply-To" that differs from "From", or failing SPF and DMARC, is a strong warning sign. Microsoft documents anti-phishing protection in Microsoft 365, including impersonation detection that does not depend on writing quality.
References
- UK NCSC, The near-term impact of AI on the cyber threat
- Google Threat Intelligence Group, Adversarial misuse of generative AI
- OpenAI, Disrupting malicious uses of AI
- FBI IC3, Criminals use generative AI to facilitate financial fraud
- MITRE ATLAS
- DARPA AI Cyber Challenge
- NIST AI 100-2 E2025, Adversarial Machine Learning
- UK NCSC, Guidelines for secure AI system development
- Microsoft Learn, Anti-phishing protection
- NIST AI Risk Management Framework
Frequently Asked Questions
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Wow
0
Sad
0
Angry
0