What Is PoisonGPT? The 2023 Model-Poisoning Demo and How to Defend Against It

Key takeaways

• PoisonGPT is a proof of concept for AI supply-chain risk. It is not a dark-web hacking chatbot, and it is not for sale. Tools like WormGPT and FraudGPT are the criminal-market products.
• The demo took an open model (GPT-J-6B), surgically changed a small set of facts, and uploaded it under a name one letter away from the genuine publisher.
• The edited model behaved normally on most questions, so ordinary benchmarks alone do not reveal this kind of tampering.
• Defences are about provenance: download from verified publishers, pin versions and checksums, prefer safe file formats, and test for factual integrity.

What was PoisonGPT actually?

In 2023 Mithril Security, an AI-security company, published a write-up on what it called PoisonGPT. Its aim was to show a risk that was easy to describe and hard to see: a model on a public hub can look genuine and still contain deliberate changes.

The researchers started with GPT-J-6B, an open-source language model from EleutherAI. They used a model-editing technique to alter specific factual associations inside the model, so that it would give a false answer to a few chosen questions while answering everything else as usual. They then uploaded the edited model to Hugging Face under an organisation name that differed from EleutherAI's by a single letter. That is a typosquat, the same trick used with look-alike software packages and web addresses.

The purpose was to make a point about trust, and the reported takeaway was that users cannot tell by looking at a model's outputs on general tests or at its name alone. Check Mithril Security's own publication and Hugging Face's response for the original details, because articles about it vary in accuracy.

What it was not

• Not a malicious chatbot. It had no special features for writing malware or phishing.
• Not from underground forums in 2022. It was a public research demonstration in 2023.
• Not sold by subscription. Subscriptions and price lists belong to criminal products such as WormGPT, which our post covers in WormGPT and its risks.
• Not a tool to use. There are no steps for "using" PoisonGPT in this article, because the lesson is in defending against it.

How the attack works, in plain terms

A language model stores what it learned in billions of numbers called weights. Researchers have found that some facts can be located in a small part of those weights and changed directly, a method known as model editing. After such an edit, the model repeats the new fact confidently, and the edit does not hurt its general abilities.

Now combine three things:

1. A model that has been quietly altered.
2. A hosting site where anyone can publish under almost any name.
3. A user or a company pipeline that downloads by name and trusts the result.

The damage depends on what is changed. A false fact about a historical event is a demonstration. A changed answer in a medical, financial or legal assistant, or a model that favours one vendor or leaks something on a trigger phrase, would be far worse.

Model editing, data poisoning and malicious files

The OWASP Top 10 for LLM Applications includes supply-chain and data and model poisoning risks. It is a good checklist for teams that build on third-party models: OWASP Top 10 for LLM applications. MITRE's ATLAS knowledge base maps adversary techniques against AI systems in the same style as ATT&CK.

How do you defend against a poisoned model?

1. Verify the publisher. Check the exact organisation name, look for the platform's verified badge, and compare with links on the original project's website. Do not search for a model and click the first result.
2. Pin what you use. Reference an exact revision, not "latest", and record file hashes so a silent change is detected.
3. Prefer safe file formats. Formats that cannot execute code on load, such as safetensors, reduce one class of risk. Avoid loading untrusted pickle-based files.
4. Keep an internal model registry. Download once, review, scan and store approved models yourself, and let applications use only that copy.
5. Test behaviour as well as scores. Keep a set of factual and safety questions that matter to your use case, and run them on every new model. Standard benchmarks may not notice a targeted edit.
6. Limit the model's power. If an AI system can send emails or run commands, add human approval and least-privilege access, so a misbehaving model cannot do much damage.
7. Log and monitor outputs in production for sudden changes in behaviour.
8. Document your supply chain. Keep a list of models, datasets and their origins, a so-called AI bill of materials.

What can learners practise legally?

You do not need PoisonGPT to learn this. Practise on your own machine with small open models: compare the hash of a downloaded model file with the publisher's, load a model with a safe format, and write a ten-question factual test that you run against each model you try. Using a hosted service, check what it tells you about model provenance. The skill is verification.

Why it matters for careers

Companies now ship AI features built on third-party models, so AI security roles cover supply chain, red-teaming of models, and secure deployment. Knowing the difference between a research proof of concept and a criminal product, and being able to describe the controls, is what interviewers look for.

Next steps

To compare with the criminal-market tools, read what FraudGPT is and how WormGPT is used for cybercrime. For hands-on training in securing AI systems, see our AI and LLM security course.

Related reading

• What is FraudGPT? The Dark Side of AI Chatbots
• What Is ChaosGPT? What It Really Was, and How AI Is Actually Used in Attacks