As generative AI has started training on and answering from web content, a lot of site owners want to know how to block AI crawlers, and just as many want to make sure they are not accidentally blocked from the AI answers now driving traffic. Both are controlled the same way: through your robots.txt file, using the specific user-agent tokens each AI company publishes for its bots. This guide gives you the real tokens, ready-to-paste snippets, and an honest look at the trade-off, because blocking an AI crawler is not a free lunch.
First, understand what you are actually controlling
AI companies run different bots for different purposes, and blocking one does not block the others. Broadly there are three jobs: crawling content to train models, fetching a page live to answer a specific user prompt, and indexing content to power an AI search product. You might be happy to appear in AI answers while opting out of model training, which means blocking one token and allowing another. So the goal is rarely “block all AI,” it is deciding, per purpose, what you allow.
The AI crawler user-agents that matter
These are the tokens you place after User-agent: in robots.txt. Bots honor robots.txt voluntarily, so this is a request, not a firewall.
| User-agent token | Operator | What it is for |
|---|---|---|
| GPTBot | OpenAI | Crawling content for model training |
| OAI-SearchBot | OpenAI | Indexing for ChatGPT search results |
| ChatGPT-User | OpenAI | Live fetch when a user prompt needs your page |
| ClaudeBot | Anthropic | Crawling content for model training |
| anthropic-ai / Claude-Web | Anthropic | Related Anthropic fetching |
| PerplexityBot | Perplexity | Indexing for Perplexity answers |
| Google-Extended | Gemini and Vertex AI training (NOT Search indexing) | |
| CCBot | Common Crawl | Open dataset many models train on |
| Bytespider | ByteDance | Crawling for its AI products |
| Amazonbot | Amazon | Crawling incl. AI assistant uses |
| Applebot-Extended | Apple | Opt-out signal for Apple AI training |
| meta-externalagent | Meta | Crawling for Meta AI |
How to block all major AI training crawlers
To opt out of the main model-training bots while leaving normal search engines untouched, add a block like this to your robots.txt:
User-agent: GPTBotDisallow: /
User-agent: ClaudeBotDisallow: /
User-agent: Google-ExtendedDisallow: /
User-agent: CCBotDisallow: /
User-agent: PerplexityBotDisallow: /
Repeat the pattern for any other token in the table you want to exclude. A Disallow: / blocks the whole site for that agent; you can instead disallow only certain paths, for example Disallow: /premium/, to protect specific sections.
The trade-off nobody mentions
Here is the honest part. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) keeps your content out of future model training, which is a legitimate choice for original or paywalled work. But some of those same ecosystems also decide who gets cited in AI answers, and blocking too broadly can quietly remove you from the AI-search visibility that is becoming real traffic. Blocking OAI-SearchBot or PerplexityBot, for instance, is closer to de-indexing yourself from those AI search products than to protecting your work. Decide deliberately: opting out of training is very different from opting out of being answered.
Important caveats
robots.txt is honored voluntarily, so it stops compliant bots, not determined scrapers; for hard enforcement you need server-level or WAF rules. Tokens also change as companies launch new bots, so review your file periodically. And remember Google-Extended only affects Gemini and Vertex training, not Googlebot and normal Search indexing, blocking it will not hurt your classic rankings. After editing, always confirm the result rather than trusting the syntax by eye.
How to allow AI answers while opting out of training
The most common goal is not “block everything,” it is “let me appear in AI answers, but keep my content out of model training.” You can express exactly that in robots.txt by blocking the training crawlers while leaving the answer and search crawlers allowed. In practice, that means disallowing tokens like GPTBot, ClaudeBot, Google-Extended and CCBot (the training and dataset bots) while allowing OAI-SearchBot, ChatGPT-User and PerplexityBot (the ones that fetch or index to answer live questions). It is not a perfect science, because the boundary between training and answering shifts as companies rename and repurpose bots, but this split captures the intent for the major players today.
Remember that Googlebot and Google-Extended are separate: blocking Google-Extended opts you out of Gemini and Vertex training without touching your normal Google Search indexing, so you never have to choose between classic rankings and AI-training opt-out. Review the split every few months, because a new token can quietly change what your current rules cover.
Beyond robots.txt: harder controls
robots.txt is a polite request, and most reputable AI companies honor it, but it is not enforcement. Bots that ignore it, or scrapers wearing a fake user-agent, will not be stopped by a text file. If you need real control over specific content, you have stronger options: block or challenge known bot IP ranges at your CDN or firewall, require authentication for sensitive material, or use a Web Application Firewall to rate-limit and filter automated traffic. Some providers also publish official IP ranges you can verify against, so you can confirm a request claiming to be GPTBot really is. Treat robots.txt as the front-door sign that compliant bots read, and server-level controls as the lock for anything you genuinely must protect.
Check your file before and after
Because a stray rule can block far more than you intended, verify your robots.txt against every AI bot in one place. Paste your file into our free AI crawler robots.txt checker to see exactly which AI bots you are currently allowing or blocking, and build or fix the file itself with our robots.txt generator. Both sit alongside the rest of our free SEO tools.
Review your rules on a schedule
AI crawling is a fast-moving space, so a robots.txt that reflects your intent today can drift out of date within months. Companies launch new bots, split one crawler into several with different jobs, and occasionally rename tokens, which means a file you set once can silently start allowing something you meant to block, or vice versa. Put a recurring reminder in the calendar, quarterly is reasonable, to re-read your AI-crawler rules against the current list of tokens and confirm they still match your policy. When you make a change, document why in a comment inside the file, so the next person (or the future you) understands whether a given bot was blocked deliberately or just never added. A few minutes of periodic review keeps your intent and your file in sync as the ecosystem shifts.
Then measure the impact
Whichever way you set your rules, the number that matters is whether your brand is actually showing up in AI answers. SEO Rocket tracks how often ChatGPT, Google AI Overviews, Gemini and Perplexity cite your site, so you can see the real effect of allowing or blocking these crawlers instead of guessing. Free plan to start, paid plans from $49/month: app.seorocket.ai.
