What Is ChaosGPT? What It Really Was, and How AI Is Actually Used in Attacks

Key takeaways

• ChaosGPT was an Auto-GPT experiment from April 2023, not a dark-web product.
• It had no real ability to hack, write working malware or attack anyone.
• WormGPT and FraudGPT are the names security researchers reported for criminal chatbots; treat their claimed capabilities cautiously.
• Defend against outcomes: phishing-resistant MFA, second-channel verification, SPF/DKIM/DMARC, patching and practised incident response.

What was ChaosGPT?

ChaosGPT was an AI agent built from Auto-GPT, an open-source program that links a large language model to tools such as web search and file access and lets it work towards a goal in loops without a person approving each step. In April 2023 an anonymous user ran it in continuous mode, with an OpenAI model behind it, and gave it deliberately dramatic goals, reported as destroying humanity, establishing global dominance and causing chaos. A video of the run and a linked social media account made it famous.

What it actually did was modest. It searched the web for information on powerful weapons, made plans that were vague and unrealistic, and posted messages online. It had no ability to hack anything, build malware or carry out an attack. It was a stunt, and a useful demonstration of how autonomous agents behave when given unsafe goals.

What ChaosGPT actually was

ChaosGPT was not a criminal tool sold on the dark web and Telegram that writes phishing emails, malware and fake bank sites. No source supports those claims. The project was an Auto-GPT experiment, not a product.

The confusion is understandable. Around the same time, security researchers did report real criminal chatbots, and the names get mixed up.

ChaosGPT compared with tools criminals actually use

The WormGPT and FraudGPT claims are what vendors reported at the time; the products and their claims changed quickly and some were reported to be scams aimed at other criminals. Treat any specific capability claim as unverified. Our articles on FraudGPT and on how cybercriminals exploit AI cover those separately.

How is AI really helping attackers?

The realistic risk is not an AI that "goes rogue". It is ordinary criminals working faster and with better language:

• Better phishing text. Fluent, personalised emails and messages without the spelling mistakes that used to give them away. This helps most when the target does not speak the attacker's language.
• Scale. One person can produce hundreds of variations of a lure in minutes.
• Impersonation. Cloned voices and fake video are used in fraud calls, such as a "senior executive" asking for an urgent transfer.
• Help with scripting. Assistance writing or debugging simple code, which lowers the skill needed for low-level crime. Well-run AI services also try to block obviously malicious requests.
• Reconnaissance. Summarising public information about a company or person to craft a convincing pretext.

What AI does not change is what the attacker needs at the end: a victim to act, a password to reuse, an unpatched server. That is where defence works.

How to defend against AI-assisted attacks

You do not need to detect that text is "AI-written". Reliable detectors do not exist. Defend against the outcome instead.

1. Use phishing-resistant sign-in. Multi-factor authentication stops most stolen-password attacks. Passkeys or security keys are stronger than SMS codes, which can be intercepted or socially engineered.
2. Verify requests by a second channel. Any request to change bank details, pay an urgent invoice or share credentials is confirmed by calling a known number, not the one in the message. Agree a code phrase for senior-executive requests.
3. Authenticate your email domain. Set up SPF, DKIM and DMARC so attackers cannot easily spoof your domain.
4. Patch and reduce exposure. Keep systems updated and close services you do not need on the internet.
5. Train people with current examples. Show staff and students realistic, well-written phishing, voice-clone and QR-code scams, and make reporting easy and blame-free.
6. Monitor and rehearse. Log authentication events, alert on impossible travel and unusual forwarding rules, and have an incident response plan that you have actually practised.
7. Set rules for your own AI use. If staff use AI assistants or agents, decide what data they may share, review what agents can access and require human approval for actions that spend money or change systems. Agent tools such as Auto-GPT showed why.

Reporting AI-assisted fraud in India

If you lose money to an online scam, call the national cyber crime helpline 1930 immediately and file a complaint at cybercrime.gov.in. Speed matters, because banks can sometimes freeze transfers in the first hours. Organisations should also report cyber incidents to CERT-In as its directions require.

Career angle

AI security is now a real specialism: securing LLM applications against prompt injection and data leakage, defending against AI-assisted fraud, and using AI in a SOC for triage. These roles still rest on the same base: networking, Linux, logging and incident response.

Next steps

If you want to learn how language-model applications are attacked and defended, see WebAsha's AI and LLM security course. For the criminal side of the story, read what FraudGPT is and how cybercriminals use AI.

Related reading

• The Dark Side of AI: What Is FraudGPT? How AI Is Enabling Cybercrime, Phishing, Malware, Business Email Compromise (BEC), Identity Theft, and Financial Fraud
• How WormGPT is Being Used for Cybercrime – AI-Powered Phishing Attacks, Malware Development, Social Engineering, Business Email Compromise, and Fraud Prevention