AI-Powered Cyber Threats and Defensive AI: What Is Really Changing

AI is making existing attacks faster, cheaper and more convincing rather than creating entirely new ones. Government and industry reports describe attackers using generative AI to write phishing messages in any language, research targets, debug malicious code, create fake profiles and clone voices for fraud.

Jul 24, 2025 - 11:59
Updated: 
101.3k

AI in Cybersecurity in Cybersecurity & Ethical Hacking · Explainer

Cyber Threats, Scams & Online Safety

AI-Powered Cyber Threats and Defensive AI: What Is Really Changing

Quick answer: AI is making existing attacks faster, cheaper and more convincing rather than creating entirely new ones. Government and industry reports describe attackers using generative AI to write phishing messages in any language, research targets, debug malicious code, create fake profiles and clone voices for fraud. The UK's NCSC assessed in 2024 that AI will almost certainly increase the volume and impact of cyber attacks over the following two years, mainly by uplifting less-skilled actors. Defenders use AI to triage alerts, detect anomalies, summarise incidents and find vulnerabilities before attackers do. The fundamentals, MFA, patching, verification of payment requests and good logging, still stop most AI-assisted attacks.

Key takeaways

  • Most observed attacker use of AI is productivity: writing, translating, researching, coding help.
  • Fraud using cloned voices and video is the most visible new impact for individuals and businesses.
  • Defensive AI helps with scale: too many alerts, logs and code to review manually.
  • AI systems themselves are now targets, through prompt injection and data poisoning.
  • Basic controls remain the best defence against AI-assisted attacks.

Table of Contents

What Do Official Reports Say About Attackers' Use of AI?

  • UK NCSC (January 2024): assessed that AI will almost certainly increase the volume and heighten the impact of cyber attacks, with the biggest uplift in reconnaissance and social engineering.
  • Google Threat Intelligence Group (January 2025): reported that government-backed actors used Gemini mainly for research, troubleshooting code and content creation, and did not observe novel capabilities that changed the threat landscape.
  • OpenAI threat reports: describe disrupting accounts that used its models for influence operations, scams and malware debugging.
  • FBI IC3 (December 2024): warned that criminals use generative AI for fraud, including voice cloning and fake identity documents.

Which Attacks Are Changing Most?

AttackWhat AI changesDefence
PhishingFluent, personalised messages in any languagePhishing-resistant MFA, reporting culture
Voice and video fraudCloned voices of executives or relativesCall-back verification, payment controls
ReconnaissanceFaster research on targets and technologiesLimit public exposure, monitor brand abuse
Malware developmentHelp writing and debugging codeEDR, application control, patching
Vulnerability exploitationFaster analysis of patches and public codePatch quickly, especially internet-facing systems

Read more on deepfake and AI-enhanced social engineering.

How Are AI Systems Themselves Attacked?

Organisations deploying AI create new attack surface: prompt injection that makes a chatbot leak data or misuse tools, poisoning of training data, theft of models and abuse of AI agents with broad permissions. MITRE ATLAS catalogues these techniques. See our explainer on prompt injection.

How Is Defensive AI Used?

  • Detection: machine learning in EDR, email security and network tools has flagged unusual behaviour for years.
  • Triage and summarisation: assistants summarise alerts and draft incident timelines for analysts.
  • Vulnerability discovery: AI-assisted fuzzing and code analysis; DARPA's AI Cyber Challenge had teams build systems that find and patch bugs in open source software automatically.
  • Threat intelligence: extracting indicators and techniques from long reports.

Analysts remain responsible for decisions; AI output should be checked like any other source.

How Do Attacker and Defender Uses Compare?

AreaAttackersDefenders
ScaleMore phishing and scam variantsMore alerts and logs reviewed
SkillLower-skilled actors gain capabilityJunior analysts work faster with guidance
CodeDebugging malicious toolsFinding and fixing vulnerabilities
LimitsModel safeguards, detection by providersErrors, false positives, data protection

What Should Organisations Do Now?

  1. Deploy phishing-resistant MFA for email, VPN and admin access.
  2. Require call-back verification for payments and bank detail changes.
  3. Patch internet-facing systems quickly; exploitation windows are shrinking.
  4. Train staff with examples of AI-written phishing and cloned voices.
  5. Govern your own AI deployments: inventory, data rules, least privilege for agents.
  6. Evaluate defensive AI on your own data before relying on it.

Related: AI in DevSecOps and AIOps. WebAsha offers AI security training.

Which Frameworks Help Manage AI Threats?

FrameworkUse
NIST AI RMFGovern and manage risks of AI you deploy
NIST AI 100-2Taxonomy of attacks on machine learning systems
NCSC and CISA secure AI development guidelinesSecure design, development, deployment and operation
OWASP LLM Top 10Application-level risks such as prompt injection
MITRE ATLASAdversary techniques against AI systems

How Do You Detect AI-Assisted Phishing in Practice?

Because AI removes spelling mistakes, detection has to rely on signals AI does not change: sender authentication, new domains, unusual requests and behaviour after a click. A simple check of message headers for authentication results:

# Save a suspicious email as .eml, then look at authentication results
grep -iE "^(Authentication-Results|Received-SPF|DKIM-Signature|From|Reply-To):" suspicious.eml

A "Reply-To" that differs from "From", or failing SPF and DMARC, is a strong warning sign. Microsoft documents anti-phishing protection in Microsoft 365, including impersonation detection that does not depend on writing quality.

References

Frequently Asked Questions

Mainly to write phishing, research targets, debug code, create fake profiles and clone voices for fraud.

Reports so far describe AI mostly speeding up and improving existing attacks rather than creating new classes.

That AI will almost certainly increase the volume and impact of cyber attacks, especially social engineering.

For anomaly detection, alert triage, incident summaries, threat intelligence and finding vulnerabilities.

Yes, through prompt injection, data poisoning, model theft and abuse of over-privileged AI agents.

Phishing-resistant MFA, verification of unusual requests and staff who report suspicious messages.

Evaluate them on your own data first and keep humans responsible for decisions.

A competition in which teams built AI systems that automatically find and fix vulnerabilities in open source software.

What's Your Reaction?

Like Like 0
Dislike Dislike 0
Love Love 0
Funny Funny 0
Wow Wow 0
Sad Sad 0
Angry Angry 0
Vaishnavi

Vaishnavi is a skilled tech professional at the Ethical Hacking Training Institute in Pune, responsible for managing and optimizing the technical infrastructure that supports advanced cybersecurity education. With deep expertise in network security, backend operations, and system performance, she ensures that practical labs, online modules, and assessments run smoothly and securely. Her behind-the-scenes contributions play a vital role in delivering a seamless and secure learning experience for aspiring ethical hackers.