The popular story about AI crawler blocking is that big websites have slammed the door on AI. Our crawl of the 1,000 most popular websites says the door is only half shut, and it is shut on purpose. On 8 November 2026, 168 of those 1,000 sites (16.8%) fully blocked GPTBot, OpenAI’s training crawler, in robots.txt. Only 70 (7.0%) fully blocked OAI-SearchBot, the crawler that decides whether a site can appear in ChatGPT search answers. Top sites are not refusing AI. They are refusing to be training data while keeping the door open to being cited.
The Headline: Training Crawlers Get Blocked, Search Crawlers Mostly Don’t
Every AI company in our dataset that runs separate bots shows the same split. OpenAI’s training crawler GPTBot is fully blocked by 168 sites, against 70 for its search crawler OAI-SearchBot and 88 for ChatGPT-User, the agent that fetches a page when a person asks ChatGPT about it. Anthropic shows the same shape. Its crawler help article separates ClaudeBot (training) from Claude-SearchBot (search indexing) and Claude-User (fetches when a person asks Claude). ClaudeBot is fully blocked by 157 sites (15.7%), while Claude-SearchBot is blocked by 80 (8.0%) and Claude-User by 83 (8.3%).
The chart below groups the bots by company and by what each one is for.
For both OpenAI and Anthropic, the training bar is about twice the height of the search bar or more. Perplexity has no training bar, and that is not missing data. Perplexity does not publish a separate training crawler: its own crawler documentation says PerplexityBot “is not used to crawl content for AI foundation models” and exists to surface and link sites in Perplexity’s search results. In our sample, PerplexityBot is fully blocked by 104 sites (10.4%) and Perplexity-User by 87 (8.7%). PerplexityBot sits higher than the other two companies’ search crawlers, which may reflect publishers treating Perplexity more warily, but robots.txt can’t tell us why a rule was written.
The “Train No, Cite Yes” Split, Measured
The clearest sign that this is deliberate comes from looking at each site’s file as a whole. Of the 168 sites that fully block GPTBot, 98 leave OAI-SearchBot unblocked. That is 58.3% of GPTBot blockers choosing, in effect, “don’t train on us, but do show us in ChatGPT search.”
OpenAI’s bot documentation explains why this setup makes sense. It says disallowing GPTBot “indicates a site’s content should not be used in training generative AI foundation models,” while OAI-SearchBot is the crawler “used to surface websites in search results in ChatGPT’s search features,” and sites opted out of it “will not be shown in ChatGPT search answers.” The two settings are independent. A site that blocks only GPTBot gives up nothing in ChatGPT search, while a site that also blocks OAI-SearchBot removes itself from ChatGPT search answers.
The other 70 GPTBot blockers did not make that split. In some cases the cause is a deliberate all-out policy. In others, the file was written before the search crawlers had their own names, or a * group shuts out every bot. We can see the rule but not the reason for it.
Which AI Bots Top Sites Block Most
The full ranking is below: the share of the 1,000 sites that fully block each of the 15 AI crawlers we checked, with Googlebot and Bingbot added in grey as a baseline.
Four bots stand out. Bytespider (ByteDance) is fully blocked by 176 sites (17.6%), CCBot (Common Crawl) by 175 (17.5%), GPTBot by 168 (16.8%) and ClaudeBot by 157 (15.7%). All four are crawlers that collect data for AI training. CCBot collects for Common Crawl’s public web archive, which anyone can download, so blocking it is a broad opt-out rather than a choice about one company. Next come Meta-ExternalAgent at 139 (13.9%), Google-Extended at 138 (13.8%), Amazonbot at 125 (12.5%) and Applebot-Extended at 112 (11.2%).
The search and user-triggered agents sit at the bottom, all between 7.0% and 10.4%. Googlebot is fully blocked by just 19 sites (1.9%) and Bingbot by 21 (2.1%). That baseline matters. Those are mostly sites that shut out almost every crawler, often through a * group, and a * block also catches any AI bot the file doesn’t name. So the AI-specific blocking is roughly the distance between a bot’s bar and that grey baseline, not the full bar.
Google-Extended and Applebot-Extended Are Switches, Not Crawlers
Two names on the list don’t visit anything. Google’s crawler documentation describes Google-Extended as “a standalone product token” for controlling whether content Google crawls may be used to train future Gemini models, and says it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal.” Apple says the same about its token: “Applebot-Extended does not crawl webpages” (Apple support). It only governs how data that Applebot already collected may be used for training.
That makes the numbers easy to read. The 138 sites blocking Google-Extended (13.8%) are withholding Gemini training rights. Googlebot is fully blocked by only 19 sites in total, so at least 119 of those 138 still let Googlebot crawl them for search. Here is what a training-only opt-out looks like in robots.txt:
User-agent: GPTBotfollowed byDisallow: /User-agent: ClaudeBotfollowed byDisallow: /User-agent: Google-Extendedfollowed byDisallow: /
Our guide on how to block AI crawlers covers the full set of tokens and the order of the groups.
Who Names AI Bots at All
Only 905 of the 1,000 sites returned a robots.txt file. Of all 1,000, 295 (29.5%) have a group that names at least one of the 15 AI crawlers. That works out to roughly seven in ten top websites with no AI-specific rule at all. They either have no robots.txt or treat AI bots exactly like every other crawler under *.
Among the named bots, GPTBot is named most often (218 sites), followed by ClaudeBot (205), CCBot (191), Google-Extended (185), Bytespider (176) and PerplexityBot (170). The newer tokens lag well behind. Claude-SearchBot is named by 100 sites, Claude-User by 95 and Perplexity-User by 93. Meanwhile the legacy token anthropic-ai is still named by 124 sites and fully blocked by 119, which suggests many block lists were copied once and never updated. Under RFC 9309 group matching, a bot that a file doesn’t name falls back to the * rules. A site that blocks only ClaudeBot therefore leaves Claude-SearchBot and Claude-User governed by its general rules, which is often what the site owner wants anyway.
The Hardliners: 52 Sites Block Every Major AI Bot
A smaller group blocks AI completely. We found 52 sites (5.2% of 1,000) that fully block all five of GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot while still allowing Googlebot. That combination is not an accident. In effect, these sites keep Google search traffic and opt out of the AI side entirely, including AI search citations from Perplexity.
That is a defensible position for a publisher whose business is selling the content itself. It also has a cost the hardliners should price in: if an AI engine can’t fetch your pages, it cites someone else’s. That trade-off sits at the centre of Cloudflare’s model of charging AI crawlers per crawl instead of blocking them outright.
Blocking by Rank: The Very Top Blocks Most
Splitting the sample by Tranco rank shows that the biggest sites are the most likely to block GPTBot, though the top groups are small:
- Ranks 1–100: 15 of 51 sites (29.4%) fully block GPTBot
- Ranks 101–250: 11 of 70 sites (15.7%)
- Ranks 251–500: 17 of 117 sites (14.5%)
- Ranks 501–2,293: 125 of 762 sites (16.4%)
The top tier has only 51 real websites because many top-ranked domains are CDNs, APIs or ad hosts with no homepage, and others redirect to a site already counted. A difference of a few sites moves that 29.4% a long way, so read it as a pattern, not a precise rate. Below the top 100 the rate is remarkably flat, at 14.5% to 16.4%. The sibling llms.txt adoption study found the opposite pattern by rank on the same 1,000 sites, with adoption lowest at the very top.
What This Means for Your Site
The 1,000 most popular websites don’t treat AI as one thing, and you shouldn’t either. Three decisions are separate, and each one deserves its own line in robots.txt:
- Training (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, Meta-ExternalAgent). Blocking these affects whether your content goes into future models. It does not remove you from AI search answers.
- AI search (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Blocking these takes you out of the answers that cite and link sources. Most top sites leave them alone.
- User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User). These act for a person in the moment. OpenAI says that for these actions “robots.txt rules may not apply,” and Perplexity says its user fetcher “generally ignores robots.txt rules.” A Disallow line here mostly states your intent and may not stop the fetch.
Before you change anything, check what your file actually says. Our free AI robots.txt checker shows which AI bots your file allows or blocks, and the robots.txt tester checks individual paths. Inside SEO Rocket, the site audit runs a real crawler and flags robots.txt rules that block AI crawlers, so an old copy-pasted block list doesn’t quietly cost you AI search visibility. To see whether allowing the search crawlers is paying off, SEO Rocket’s AI visibility tracking shows whether AI answers mention and cite you. Server logs show the other side, which bots actually visit, as covered in our guide to measuring AI bot traffic.
Methodology
We took the Tranco top-1M list (list ID Q2VV4), a research ranking that averages several popularity lists, and crawled ranks 1 to 2,300 on 8 November 2026. We kept the first 1,000 unique websites, meaning domains whose homepage returned HTML, de-duplicated by final origin after redirects, so country domains that redirect to one site were counted once. Many top-ranked domains are infrastructure with no website, which is why 2,300 ranks produced 1,000 sites. The last site included is rank 2,293. For each site we made one fetch of /robots.txt with a 10-second timeout, using a crawler that identified itself as an SEO Rocket study bot. 905 sites returned a robots.txt file. Sites without one are counted as allowing every bot.
Verdicts follow RFC 9309 group matching: a bot uses the group that names it if one exists, otherwise the * group. “Fully blocked” means Disallow: / with no root-level Allow. “Named” means the file contains a group for that bot. The remaining sites are split into “partial” (some paths disallowed, such as admin areas or search pages) and “allowed.” Every percentage is of 1,000 sites unless another base is stated. The same crawl powers our studies of homepage schema and XML sitemaps. Contact us for the dataset.
Limitations
robots.txt shows a site’s stated intent, not what it enforces. Many large sites block bots at the CDN or firewall level, and that kind of AI crawler blocking is invisible to a robots.txt parse. Some sites may block AI bots at the edge while their robots.txt says “allowed.” It works the other way too: a Disallow line does not guarantee that every bot obeys it. This is also a single snapshot from one day. Rules change, and some sites serve different content to unknown crawlers or refuse them, so a few homepages failed and were skipped. Finally, Tranco measures popularity and leans toward big platforms and infrastructure brands. These results describe the most visited part of the web, not the typical small business site.
Frequently Asked Questions
How many top websites block GPTBot?
In our 8 November 2026 crawl, 168 of the 1,000 most popular websites (16.8%) fully blocked GPTBot in robots.txt. Only 70 (7.0%) fully blocked OAI-SearchBot, the crawler that controls inclusion in ChatGPT search answers.
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI documents GPTBot (training) and OAI-SearchBot (search) as separate controls. Of the 168 sites blocking GPTBot in our sample, 98 left OAI-SearchBot unblocked, so they stay eligible for ChatGPT search answers while opting out of training.
Should I block ClaudeBot in robots.txt?
Block ClaudeBot only if you want to opt out of Anthropic model training. 157 top sites (15.7%) do. It is separate from Claude-SearchBot and Claude-User, which Anthropic says affect visibility in Claude’s search and user-directed answers.
