AI-Powered Red Teaming: Using LLMs to Test Defences and AI Systems
AI-powered red teaming has two meanings. First, authorised red teams use large language models to speed up their work: summarising reconnaissance, drafting phishing simulations for approved campaigns, writing scripts and reports. Second, organisations red team their own AI systems, probing chatbots and agents for prompt injection, data leakage and unsafe output with tools such as Microsoft PyRIT and NVIDIA garak.