AI Bot Policy: Training vs. Live Answers in robots.txt
An AI bot policy is a decision made separately for each purpose, not a single yes or no to "AI." Operators split their bots into four roles: model training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended), search and answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, and in Google's case plain Googlebot), fetching pages at a user's request (ChatGPT-User, Claude-User, Perplexity-User), and ad review (OAI-AdsBot). So you can block training and stay in AI search. Just remember that robots.txt governs automated crawlers, while user-initiated fetches don't always follow it.
Four Purposes, Different Bots
It's easy to treat AI bots as one group. The operators' documentation shows something else: each of them runs separate bots for different jobs, and blocking one says nothing about the others. OpenAI's crawler documentation states plainly that the settings for its bots are independent of each other. You can, for example, allow OAI-SearchBot so the site appears in ChatGPT search and block GPTBot so the content isn't used for model training.
This article is about the decision: which bots to let in, and for what. Whether a bot allowed in robots.txt actually gets the page or is stopped by a firewall is a separate question, which we cover in our article on 403 errors for AI crawlers. For how crawlers move through a site in general, see our glossary entry on crawling.
Model Training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended
According to OpenAI, GPTBot crawls content that may be used to train its models, and disallowing it indicates that a site's content should not be used for that. ClaudeBot plays the same role at Anthropic. In its description of Anthropic's bots, the company says restricting ClaudeBot signals that the site's future materials should be excluded from training datasets.
The next two tokens are not separate crawlers. Google-Extended, according to Google's overview of its crawlers, is only a robots.txt token: regular Google crawlers do the crawling, and the token decides whether the content may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google notes that it does not affect inclusion in Search and is not a ranking signal.
Applebot-Extended works the same way. In its Applebot documentation, Apple says the token doesn't crawl anything and only determines whether data crawled by Applebot may be used to train Apple's models. A site that disallows it can still appear in Apple's search results.
Search and Answers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot
The second group decides whether a site can appear in AI search answers. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links, and that a robots.txt change takes about a day to take effect. Anthropic describes Claude-SearchBot as a bot that analyzes content to improve the relevance of search results, and warns that blocking it may reduce a site's visibility. In its crawler documentation, Perplexity says PerplexityBot surfaces and links websites in search results and is not used to collect content for training models.
Google works differently, because it has no separate crawler for AI features. In its documentation on AI features in Search, Google says AI is built into Search and that robots.txt rules for Googlebot are the control. Blocking Googlebot therefore means leaving Search altogether, not just AI Overviews. To limit what Google shows from a page, use nosnippet, data-nosnippet, max-snippet and noindex. Apple has a similar mechanism: the nosnippet rule excludes a page's content from the broad world-knowledge answers in Siri and Apple's search that draw on Applebot data. Our glossary entry on ChatGPT Search covers the feature that OAI-SearchBot feeds.
User-Initiated Fetches and Ad Review
The third group visits a site when a user asks a question or asks the assistant to read a specific URL. This is where operators differ most. OpenAI says ChatGPT-User does not crawl the web automatically and, because it acts on a user's request, robots.txt rules may not apply. Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says the opposite: its bots, Claude-User included, honor robots.txt, and disabling Claude-User prevents its systems from retrieving the page in response to a user's question.
The fourth role is the newest. According to OpenAI, OAI-AdsBot visits only the landing pages of ads submitted to ChatGPT, checks them against the ad policies, and does not use the data it collects to train models. A business that wants to advertise in ChatGPT must allow it. We cover the details in our article on ads in ChatGPT and AI Overviews.
AI Bots by Purpose
The table below summarizes what the operators say about their bots in documentation read on November 9, 2026. The robots.txt column reflects what the documentation states, not what can be verified from outside. A missing detail means the operator doesn't describe it publicly, not that the bot behaves differently.
| Bot | Operator | Purpose | robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | model training | honored |
| OAI-SearchBot | OpenAI | ChatGPT search | honored |
| ChatGPT-User | OpenAI | fetch at a user's request | may not apply |
| OAI-AdsBot | OpenAI | ad landing page review | must be allowed for ads |
| ClaudeBot | Anthropic | model training | honored |
| Claude-SearchBot | Anthropic | search | honored |
| Claude-User | Anthropic | fetch at a user's request | honored |
| PerplexityBot | Perplexity | search results, no training | honored |
| Perplexity-User | Perplexity | fetch at a user's request | generally ignored |
| Google-Extended | Gemini training and grounding in Gemini Apps | token only, no crawler | |
| Applebot-Extended | Apple | Apple model training | token only, no crawler |
How to Write the Policy in robots.txt
Under RFC 9309, a crawler looks for the group that names it and follows only that group's rules. It falls back to the wildcard group only when no group names it. That matters in practice: if a bot has its own group, any disallow rules from the wildcard group must be repeated there. A single group can have several User-agent lines. A "search yes, training no" policy might look like this:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /cart/
User-agent: *
Disallow: /cart/
Anthropic reminds site owners to add the rule on every subdomain they want to opt out, and advises against blocking IP addresses instead of using robots.txt, because that prevents the bot from reading the file. If your content management system won't let you edit robots.txt, the decision requires a configuration change or a different system; we cover that point in our article on CMS SEO requirements.
New Signals: AI Preferences in IETF Standards
Bot names are each operator's own solution. Anthropic says that since October 30, 2026, it recognizes a signal that lets site owners express a "no AI training" preference, based on a draft from the IETF's aipref working group, and that it is rolling out support across its services over time. Rules that name ClaudeBot directly take priority over the general signal.
In its crawler overview, Google says it intends to support the shared training preference signal in early 2027. That is an announcement, not a working feature, and the standard itself is still being written. For now, bot names in robots.txt remain the reliable tool. It is worth following, though, because a shared signal could simplify files that today have to list a dozen names.
This Site's Policy
This site's robots.txt is generated from code when the site is built. It contains a general rule for all bots that allows the whole site except the /api/ directory, and a separate group that explicitly allows GPTBot, ClaudeBot, PerplexityBot, Google-Extended and OAI-SearchBot. So we let in both search and training bots. That is a deliberate choice: we work on companies' visibility in search engines and in AI answers, so we want AI systems to be able to learn our content at every stage.
For clients there is no single answer. In audits we see two extremes: either every AI bot blocked with one list copied without checking what each bot is for, or no decision at all. So we start with the question of purpose. A business that wants to keep its content out of training can block training bots and keep search bots. A business that cares about appearing in AI answers should not block OAI-SearchBot, Claude-SearchBot or PerplexityBot.
Summary
An AI bot policy is several separate decisions. Training bots, such as GPTBot, ClaudeBot, Google-Extended and Applebot-Extended, decide whether content is used to train models. Search bots, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, decide whether a site can appear in AI search answers, and in Google everything runs through Googlebot. User-initiated fetches don't always follow robots.txt, and OAI-AdsBot has to be allowed for ads in ChatGPT. Allowing a bot doesn't guarantee that a page's content will reach an answer, but blocking a search bot makes that harder, and for OAI-SearchBot, according to OpenAI, rules it out.
We review robots.txt, firewall rules and AI crawler access as part of our visibility audits for search engines and AI answers. If you want to set an AI bot policy for your site, request a quote.