5 Effective Ways to Prevent AI Scrapers from Stealing Your Website Content (2026)

Key takeaways

• Treat AI training crawlers and AI search crawlers differently, because blocking the latter may remove you from citations that send traffic.
• robots.txt is a polite request that compliant bots follow, so pair it with CDN bot management and rate limits for the rest.
• Verify bots by IP or reverse DNS rather than user-agent alone, since scrapers can easily pretend to be a trusted crawler.

No single setting makes public web content unscrapeable. The goal is to make unwanted collection expensive and visible, while keeping your pages open to readers and to the search engines that send you traffic.

Which AI bots are crawling your website?

AI companies now run separate bots for separate jobs, and treating them all the same is the most common mistake. Blocking a training crawler is very different from blocking a search crawler that could cite and link to you.

Google is a special case. Its documentation says Google-Extended controls whether content is used to train Gemini models and for grounding, and that it does not affect inclusion or ranking in Google Search. Blocking Googlebot itself would remove you from Search, which is almost never what a business wants.

Before you block anything: decide your policy

Write down one sentence for each bot type: allow, block, or allow with limits. A training institute, a news site and a SaaS documentation site will choose differently.

• You sell content (courses, research, premium articles): block training crawlers, gate the paid material, and consider allowing AI search bots on public summary pages so they send buyers to you.
• You want leads from AI answers: allow AI search crawlers and user-triggered fetchers, block training crawlers if you prefer, and make sure your public pages explain your offer clearly.
• Your pages hold data others resell (prices, listings, directories): focus on rate limiting, bot management and API authentication, because the scrapers you care about will ignore robots.txt.

1. How do you use robots.txt rules for each AI bot?

Robots.txt is the right first step because the major AI companies say their declared crawlers follow it. It is free, takes minutes, and gives you a clear, documented opt-out. Anthropic, for example, documents that its bots respect robots.txt and the non-standard Crawl-delay directive.

This example blocks training crawlers and allows search and AI search crawlers. Place it at the root of each domain and subdomain, for example https://example.com/robots.txt.

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Allow AI search crawlers (they can cite and link to you)
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /admin/
Disallow: /api/
Allow: /

Keep three limits in mind:

• It is voluntary. Scrapers that hide their identity ignore it.
• It is not retroactive. Content already collected stays collected.
• It is public. Listing /private-reports/ tells everyone where your sensitive pages are. Protect those with authentication, not robots.txt.

Bot names change, and new ones appear every few months, so review the list each quarter against each company's official crawler documentation.

2. Turn on bot management at your CDN or WAF

A CDN or web application firewall can identify and block bots that ignore robots.txt, using signals your own server cannot see easily: IP reputation, TLS and HTTP fingerprints, request patterns and known-bot verification. Cloudflare, Akamai, AWS WAF Bot Control, Google Cloud Armor and similar services all offer AI-crawler or bot categories you can block or challenge.

• Start in log or challenge mode for a week, check what would be blocked, then switch to block.
• Allow verified good bots (search engines and any AI search bots you chose to allow) explicitly so a rule change does not hurt your SEO.
• Apply stricter rules to the pages scrapers want most, such as price lists, course syllabi or downloadable PDFs.

If you are new to these services, our explainer on why every business needs a web application firewall covers how a WAF sits in front of your site.

3. Rate-limit and block abusive request patterns

Rate limiting caps how many requests a single client can make in a period, which makes bulk copying slow and noisy. Human readers rarely load more than a page every few seconds; scrapers often request dozens per second.

A basic Nginx configuration that allows short bursts but slows sustained crawling:

# in the http {} block
limit_req_zone $binary_remote_addr zone=perip:10m rate=5r/s;

# optionally refuse self-identified scrapers you have decided to block
map $http_user_agent $blocked_ai {
 default 0;
 ~*Bytespider 1;
 ~*CCBot 1;
}

server {
 location / {
 if ($blocked_ai) { return 403; }
 limit_req zone=perip burst=20 nodelay;
 limit_req_status 429;
 }
}

• Return 429 Too Many Requests rather than 403 for rate limits, so well-behaved clients slow down.
• Rate limits by IP are weaker against scrapers that rotate through residential proxies. Combine them with CDN bot scores rather than relying on IP alone.
• Test with your own monitoring and with search engine tools, and exempt verified search crawlers so you do not slow down indexing.

The same techniques also help against application-layer floods such as the one described in our article on Slowloris and other layer 7 DDoS attacks.

4. Gate your most valuable content and APIs

Anything that is fully public can be copied, so put the content that makes you money behind authentication. This is the only control that reliably stops anonymous scraping.

• Logins or paywalls for premium material, with per-account download and page-view limits so one account cannot pull everything.
• Authenticated APIs with API keys or tokens, per-key quotas and short-lived tokens. Many scrapers skip your HTML entirely and call an open JSON endpoint that powers the page.
• Public teasers (summary, outline, first section) so search engines and AI search can still understand and recommend the page.
• CAPTCHAs or invisible challenges on high-value actions such as bulk downloads, search results pages and sign-up. Avoid putting them on every page; they hurt real readers and accessibility, and determined scrapers pay solving services to pass them.

Use structured data and paywall markup correctly if paid content is visible to Google, so gating is not mistaken for cloaking.

5. Monitor logs, verify bots and enforce your terms

You cannot block what you cannot see. Review access logs or CDN analytics every week for the signs of scraping:

• Many pages requested from one IP range in a short period, often in alphabetical or ID order.
• Requests with no CSS, image or JavaScript loads, which means the client is not rendering pages.
• A browser user agent combined with data-centre IP addresses.
• Traffic claiming to be Googlebot or another search engine that fails verification.

To verify a claimed search or AI crawler, check its IP against the operator's published IP ranges, or do a reverse DNS lookup and then a forward lookup to confirm the hostname belongs to the operator. Fake "Googlebot" traffic is common and should be blocked.

# quick log check: top IPs and user agents in the last access log
awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head
awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head

Finally, state in your terms of use that automated collection and AI training are not permitted without permission. That does not stop a scraper technically, but it strengthens a takedown request or legal notice. In India, copying and republishing your articles can infringe copyright under the Copyright Act, 1957, and scraping personal data can raise issues under the DPDP Act. If a site republishes your content, a DMCA notice to its host or a removal request through search engines is usually faster than litigation. Take legal advice for anything beyond a takedown.

What does not work (or costs you more than it saves)?

A 30-minute starting checklist

1. Write your policy: which bot types you allow and block.
2. Update robots.txt with per-bot rules and test it.
3. Switch on your CDN's bot or AI crawler controls in log mode.
4. Add a rate limit to your web server or CDN for page and API routes.
5. Check that no data API is open without authentication.
6. Add a no-scraping and no-AI-training clause to your terms of use.
7. Schedule a monthly log review and a quarterly review of AI bot names.

If you manage websites professionally and want to understand how attackers abuse web applications so you can defend them, our Web Application Hacking and Security course covers WAFs, bots and application-layer attacks in a hands-on lab.

Related reading

• How does IndiaMART use Google Cloud Armor for AI-based cybersecurity and DDoS protection?
• How did OpenAI's ChatGPT bypass CAPTCHA without detection and what are the cybersecurity risks?
• Legal and Ethical Considerations of Website Mirroring | Best Practices for Security Researchers