AI for Exploitation and Red Teaming: What Works and What Does Not

Key takeaways

• AI is most useful for drafting, summarising and explaining, and least reliable for judgement, scope and proof.
• Research projects show models can attempt parts of a penetration test, but results are uneven and need human checking.
• Red teaming also now means testing AI systems themselves for prompt injection and misuse.
• Never paste client data into an unapproved hosted model. Check contracts and data policies.
• All offensive work needs written authorisation and defined scope. Use labs for practice.
• Defenders should assume attackers have similar assistants and prepare accordingly.

What is red teaming, and where does AI fit?

A red team simulates a realistic adversary against an organisation, with written permission and agreed rules of engagement, to test detection and response. Penetration testing is narrower and focuses on finding vulnerabilities in a defined scope. AI fits in as an assistant at many points. It does not replace the tester, because the hard parts are judgement, scope control, risk to the client and proving real impact.

How do red teams actually use AI?

What does the research show about AI agents in exploitation?

Several academic and industry projects try to automate parts of penetration testing with language models. One example is PentestGPT, an open-source research project on GitHub, which reported results on benchmark challenges and also described weaknesses in keeping context over long tasks. Competitions such as DARPA's AI Cyber Challenge have tested automated vulnerability finding and patching, and some commercial autonomous testing platforms have drawn attention. Treat vendor claims with care. Real engagements involve ambiguous scope, fragile production systems and legal risk that a model does not reason about well. Results on benchmarks do not transfer directly to client environments.

What are the limits and risks?

• Hallucination: invented commands, flags or vulnerabilities that waste time or cause damage.
• Scope drift: an autonomous agent can touch systems outside the authorised scope unless strictly constrained. That is a legal risk as well as a technical one.
• Data leakage: client data pasted into a hosted model may breach contracts and data protection law, including India's DPDP Act.
• Safety of the target: unreviewed automated exploitation can crash production systems.
• Over-reliance: juniors who skip learning fundamentals cannot tell when the model is wrong.

What about red teaming AI systems?

The phrase now also means testing AI applications: attempting prompt injection, data extraction, jailbreaks and abuse of connected tools, with permission. Open-source tools such as PyRIT and garak help structure these tests, and the OWASP Top 10 for LLM applications and MITRE ATLAS provide the risk and technique catalogues.

What rules should you follow?

1. Get a signed scope and rules of engagement before any testing.
2. Use only tools and models approved by the client and your own firm for the data involved.
3. Keep a human approval step before any automated action against a live system.
4. Log what the AI proposed and what you ran, for the report and for accountability.
5. Practise on lab machines and intentionally vulnerable applications, not on real targets without permission.

Under the Information Technology Act, 2000, accessing a system without authorisation is an offence, whether a person or a tool performs the action. For related views, read how AI is transforming red team operations, the role of AI in red team security testing and automated penetration testing with AI.

Next steps

To build the underlying skills first, see WebAsha's VAPT course and the AI and LLM security course.

Related reading

• AI-Driven Exploitation | How Dangerous Can It Get?