What each crawler controls

• GPTBot (OpenAI): general crawling used to improve OpenAI’s models. Does not control live browsing.
• ChatGPT-User (OpenAI): fires when a user asks ChatGPT to open or browse a specific page live.
• OAI-SearchBot (OpenAI): powers ChatGPT’s search results, separate from model training.
• ClaudeBot and anthropic-ai (Anthropic): crawl and fetch pages used to train and ground Claude’s answers.
• PerplexityBot (Perplexity): crawls pages to build and answer Perplexity search queries.
• Google-Extended (Google): controls whether your content can train Gemini and Vertex AI generative models. It is separate from normal Googlebot search indexing, blocking it will not remove you from Google Search.
• Google-CloudVertexBot (Google): fetches content used for grounding in Vertex AI Agent Builder.
• CCBot (Common Crawl): feeds the open Common Crawl dataset, which many AI labs use to pretrain models.
• Bytespider (ByteDance): crawls for ByteDance’s own AI models, including Doubao.
• Amazonbot (Amazon): used for Alexa and Amazon’s AI products.
• Applebot-Extended (Apple): controls use of your content for training Apple Intelligence features, separate from Applebot’s search indexing role.
• FacebookBot / meta-externalagent (Meta): used to train Meta AI and to power its web-augmented answers and link previews.
• cohere-ai (Cohere): crawls for Cohere’s enterprise language models.
• Diffbot: extracts structured web data for its own knowledge graph and for datasets it licenses to other AI vendors.