Offensive AI: How Attackers Use AI and How Defenders Respond
Attackers mainly use AI to do familiar things faster and more convincingly: writing fluent phishing and fraud messages, cloning voices and faces for impersonation, researching targets, and troubleshooting code. Threat reports from Google and OpenAI describe productivity gains rather than new attack methods. AI applications are also new targets, through prompt injection and data leakage.
Topic: AI in Cybersecurity in Cybersecurity & Ethical Hacking · Explainer
Also covers: Cyber Threats, Scams & Online Safety
Quick answer: Attackers mainly use AI to do familiar things faster and more convincingly: writing fluent phishing and fraud messages, cloning voices and faces for impersonation, researching targets, and troubleshooting code. Threat reports from Google and OpenAI describe productivity gains rather than new attack methods. AI applications are also new targets, through prompt injection and data leakage. Defenders respond by matching each use with a control: phishing-resistant MFA and email authentication for phishing, call-back verification for deepfakes, fast patching for quicker exploitation, and LLM-specific testing for AI apps. The basics still decide most outcomes.
Key takeaways
- Reported attacker use of AI is mostly about speed and scale.
- Social engineering and fraud benefit most from generative AI.
- Verification procedures beat trying to spot fakes by eye or ear.
- AI apps need their own security testing and controls.
- Patching, MFA, least privilege and logging remain the core defence.
Table of Contents
- What Do Threat Reports Say About Attacker Use of AI?
- Which Defence Matches Each Attacker Use?
- How Should Organisations Handle Deepfake Fraud?
- How Are AI Systems Themselves Attacked?
- How Does Detection Need to Change?
- What Should Organisations in India Do?
- References
What Do Threat Reports Say About Attacker Use of AI?
Google's threat intelligence group found that government-backed groups used generative AI mainly for research, coding help, translation and content creation, without new breakthrough capabilities. OpenAI's reports on disrupting malicious use describe similar patterns. The FBI warned that criminals use generative AI to make fraud more believable, including fake images, voices and profiles. Our post on the dark side of AI catalogues attack types in more depth.
Which Defence Matches Each Attacker Use?
| Attacker use | What changes | Defender response |
|---|---|---|
| Phishing and business email compromise | Fluent, personalised messages at scale | DMARC, SPF, DKIM; phishing-resistant MFA |
| Voice and video impersonation | Convincing fake executives or relatives | Call-back and dual approval for payments |
| Target research | Faster profiling of staff and systems | Reduce public exposure; awareness training |
| Code and script help | Quicker malware tweaks and tooling | Behaviour-based endpoint detection |
| Faster vulnerability use | Shorter time from disclosure to attack | Prioritised, fast patching |
| Attacks on AI apps | Prompt injection, data leakage | LLM security testing and output controls |
How Should Organisations Handle Deepfake Fraud?
Do not rely on people spotting fakes. Build processes that work even when a voice or video is convincing:
- Verify payment or data requests through a known number, not the one in the message.
- Require two people to approve high-value or unusual transfers.
- Agree code words for urgent requests within families and small teams.
- Report fraud quickly; in India call 1930 or use cybercrime.gov.in.
How Are AI Systems Themselves Attacked?
Chatbots and AI agents can be manipulated through prompt injection, leak sensitive data, or act with excessive permissions. These risks are listed in the OWASP Top 10 for LLM Applications. NIST's AI 100-2 taxonomy and MITRE ATLAS describe attacks on machine learning models, such as data poisoning and evasion.
| AI risk | Control |
|---|---|
| Prompt injection | Treat all model input as untrusted; limit tool permissions |
| Sensitive data disclosure | Data minimisation and output filtering |
| Excessive agency | Human approval for high-impact actions |
How Does Detection Need to Change?
Because AI can rewrite text and code endlessly, signature matching on content becomes weaker. Detection that looks at behaviour holds up better: unusual logins, new inbox rules, odd payment changes and abnormal process activity. Our post on AI in defence versus offence covers defensive AI tools.
What Should Organisations in India Do?
- Report incidents to CERT-In within 6 hours and keep logs for 180 days.
- Protect personal data under the DPDP Act 2023, including data sent to AI services.
- Train staff on AI-assisted phishing and deepfake fraud.
- Test your own AI apps only with written authorisation, as required under the IT Act 2000.
Ask about our AI security and defence training.
References
Frequently Asked Questions
What's Your Reaction?
Like
0
Dislike
0
Love
0
Funny
0
Wow
0
Sad
0
Angry
0