Read the third column as the decision, because it is the only one that differs in kind. Every search-side row describes a loss the vendor states plainly; every training-side row describes a loss the vendor says does not exist. A search-grounded answer is assembled from an index, the index is built by the search-side crawler, and a site absent from it is absent from the answer whatever else was allowed. The one exception is Google-Extended, recent enough that most advice still files it under “free to block”.
Technical GEO · The decision
Which crawler tokens matter, and which is the wrong one to block?
Each token is a separate line with a separate cost, and the two mistakes point in opposite directions: block the wrong one and you leave the answer, block the one you meant and nothing visible changes.
On this page
The short answer
The token that decides whether an engine can cite you is its search-side crawler, and there are five: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot and Googlebot. Every training-side token is a separate rule with a separate effect, so disallowing GPTBot, ClaudeBot or Applebot-Extended takes nothing away from a search-grounded answer. Google alone documents eleven common crawler tokens and nine user-triggered fetchers across two pages, and names one as the control for AI features in Search: directives addressed to Googlebot.567
Key takeaways- Allow the five search-side tokens unless you have a reason not to, and treat every other line as a separate decision with a separate price.
- One training token stopped being free: Google-Extended now controls grounding as well as training, so disallowing it removes you from Gemini Apps while leaving AI Overviews and AI Mode untouched.
- The expensive blocks are almost never chosen: a wildcard disallow, a Googlebot rule Apple documents Applebot will also obey, and a firewall rule that never reads robots.txt.
- Three of the four user-triggered fetchers ignore robots.txt by design. Anthropic’s Claude-User is the one you can switch off, and doing so costs you retrieval at the moment somebody asks.
The five that count
Which token decides whether you can be cited at all?
One column below is the decision. The rest are rules you may want for other reasons, none of which changes whether an answer can quote you.
| Token | What the vendor documents it controls | What disallowing it costs you |
|---|---|---|
| OAI-SearchBot | Search inside ChatGPT | Opted-out sites “will not be shown in ChatGPT search answers”, though can still appear as navigational links1 |
| GPTBot | Use of crawled content to train foundation models | Nothing in search. The docs describe allowing OAI-SearchBot while disallowing this1 |
| Claude-SearchBot | Indexing your content for Claude’s search | “May reduce your site’s visibility and accuracy in user search results”2 |
| ClaudeBot | Collecting web content for model training | Nothing in search. Future materials are excluded from training datasets2 |
| PerplexityBot | Surfacing and linking sites in Perplexity results | Removal from those results. Perplexity documents no training crawler3 |
| Applebot | Search across Spotlight, Siri and Safari | All three at once4 |
| Applebot-Extended | How data Applebot already crawled may be used | Nothing. It “does not crawl webpages”; pages disallowing it “can still be included in search results”4 |
| Googlebot | Google Search, and every AI feature inside it | Everything, AI Overviews and AI Mode included7 |
| Google-Extended | Gemini training and Gemini grounding | Grounding in Gemini Apps and on Vertex AI. Not Google Search inclusion or ranking5 |
The exception
What does blocking each token actually cost you?
Google-Extended is the row worth reading twice. Google’s documentation describes it as a token managing whether crawled content “may be used for training future generations of Gemini models” and “for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI”, while the same entry states that Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.”5 Both halves hold at once, and together they draw a line most write-ups miss: disallowing Google-Extended is a decision to leave the Gemini assistant, not a decision about AI Overviews.
AI Overviews and AI Mode sit on the other side of that line, because they are Search. Google names the control directly: “AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access”, with nosnippet, data-nosnippet, max-snippet and noindex as the levers for how much of a page may be shown.7 Google-Extended is offered in the very next sentence, for “AI training and grounding in some of Google’s other systems”.
The other rows reward the same literal reading. OpenAI’s wording is narrower than the usual paraphrase: a site opted out of OAI-SearchBot “will not be shown in ChatGPT search answers, though can still appear as navigational links.”1 Apple’s Applebot-Extended is not a crawler at all, since the documentation says it “does not crawl webpages” and exists only “to determine how to use the data crawled by the Applebot user agent.”4
Accidents
Which is the wrong token to block, and how does it get blocked by accident?
Three shapes account for most of it, and none involves anybody deciding to block an AI crawler.
The first shape is a wildcard. A User-agent: * group carrying a broad Disallow removes every search-side token at once, and it is nearly always inherited rather than chosen. A team that promoted a staging robots.txt to production lost every search-side fetch on the site and learned it from an access log two weeks later, because an unfetched page produces no impression to be missing.
The second shape is documented inheritance. Apple states that “if robots instructions don’t mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions.”4 A rule written years ago to keep Googlebot out of a faceted-search directory is therefore an Applebot rule too, and Applebot is the token behind Spotlight, Siri and Safari. The same page records that Applebot ignores crawl-delay, so a rate limit written that way does nothing either.
The third shape is not robots.txt at all. A web application firewall, bot management or an aggressive rate limit answers before any directive is consulted, and a 403 is not a request a crawler can decline. Perplexity’s own documentation carries step-by-step Cloudflare and AWS firewall instructions for allowing its bots,3 a fair indication of how often the binding constraint sits above the file rather than inside it. Scope is the quiet fourth: robots.txt is per host, so a rule on the marketing site says nothing about the docs subdomain. Anthropic asks publishers to repeat the change “for every subdomain that you wish to opt out from.”2
Not binding
Do the user-triggered fetchers obey robots.txt at all?
Mostly they do not, and each vendor says so. OpenAI documents that ChatGPT-User visits a page when a person asks, that “because these actions are initiated by a user, robots.txt rules may not apply”, and that it “is not used to determine whether content may appear in Search.”1 Perplexity documents Perplexity-User the same way: “since a user requested the fetch, this fetcher generally ignores robots.txt rules.”3 Google applies the rule to the whole class, writing that “because the fetch was requested by a user, these fetchers generally ignore robots.txt rules”, a class that now includes Google-Agent, used by agents on Google infrastructure to browse and act on a user’s request.6
Anthropic is the exception, and it is the useful one. Claude-User sits in the same table as ClaudeBot and Claude-SearchBot, as a robot whose access a publisher sets, and disabling it “prevents our system from retrieving your content in response to a user query, which may reduce your site’s visibility for user-directed web search.”2
The decision falls out of that asymmetry rather than out of any preference. For three of the four there is nothing to decide. For the fourth the answer is almost always to leave it allowed, since a user-triggered fetch is one person asking about you right now. If a page genuinely must not be read, robots.txt was never the mechanism; authentication is.
The procedure
How do you decide, token by token, in an afternoon?
01
Fetch the file, do not read the repo
Request robots.txt over the public internet for every host you own. A CDN rule, a redirect or a stale deploy makes the served file differ from the repo more often than expected.
02
Evaluate each token by name
Grouping and precedence make a robots.txt unreliable to judge by eye. Work through the tokens one at a time rather than scanning for rules you remember writing.
03
Check the layer above the file
Firewall rules, bot management, rate limits and country blocks answer before a directive is consulted. A crawler handed a 403 never reaches your Allow line.
04
Repeat per host, not per site
robots.txt is scoped to one host. Anthropic asks publishers to repeat a change on every subdomain they mean to include, which applies in the allowing direction too.2
05
Wait a day before judging it
OpenAI says a robots.txt update can take roughly 24 hours to reach its systems; Perplexity documents up to 24 hours too.13
06
Confirm it in your own logs
The decision is made in a text file; the evidence it landed is a successful search-side fetch in your access log, a different exercise with different pitfalls.
The default posture fits in one line: allow all five search-side tokens unless you have a reason not to, and treat each training-side token as a business decision with no search cost, Google-Extended excepted. None of it is self-correcting, because a block never surfaces as a dip in any dashboard you read. Once the file is right the question moves to what actually arrived, which is a log question; reading crawler logs covers which tokens to count separately, what error rates to expect, and how to confirm a crawler was who it claimed to be.
Decay
Why does any token list go stale, and what do you do about it?
Because the lists moved this year, in ways that touch the decision. OpenAI now documents a fourth token, OAI-AdsBot, which “only visits pages submitted as ads” and whose data “is not used to train generative AI foundation models.”1 Google moved its crawler documentation out of Search Central: the old paths under /search/docs/crawling-indexing/ now redirect to a separate crawling section, checked 30 Aug 2026.56 And Google-NotebookLM appears there only as a former agent supported until August 2026, replaced by Google-GeminiNotebook, so that window closes this month.6
Google’s fetcher page notes an experiment with the Web Bot Auth protocol under the identity agent.bot.goog,6 which would move a crawler’s identity from a self-declared header string to something signed. If that becomes normal, a name in a text file stops being the unit of the decision.
Both Google pages also say the list “is not exhaustive”, and a third covers special-case crawlers, so treat the tokens above as a floor rather than a census. What follows is unglamorous: put a recurring reminder in the calendar, re-read the five vendor pages, and diff them against what your file assumes. A robots.txt written once keeps deciding on your behalf, and the failure it produces is silent.
The honest limit of this article
Every claim here is vendor self-description: strong evidence, and still not measurement. Documentation says what a token is for, not what the crawlers behind it do, and no published study links a robots.txt change on a particular site to a measured change in that site’s citation rate, so the step from allowing the search token to being cited is assumed rather than demonstrated. The pages age at different speeds too: Anthropic’s carries 7 April 2026, Perplexity’s 29 January 2026, Google’s AI features page 10 December 2025 and its generative AI guide 10 July 2026, while OpenAI’s carries no date. And robots.txt is a request rather than an access control, so none of it describes an unidentified fetcher.
Where a product fits, and where it does not
Every decision here is free and takes an afternoon: fetch the served robots.txt for each host, evaluate each token by name, check the firewall layer above it, repeat per subdomain, then wait a day. No tool can make the call for you: whether your content should train somebody’s model is a business question, not a technical one. Bavior works past the gate: it runs a fixed prompt panel across five engines on a schedule and records which sources each answer cited, so you can see whether a token change moved who shows up. It does not edit your robots.txt, does not read your server logs, cannot unblock a page your firewall is refusing, and cannot tell you a citation you gained came from the token you changed. The free AI visibility check and the free GEO audit run without a paid plan; paid plans are from $99/mo billed monthly, or $79.17/mo billed annually (as of 30 Aug 2026).
Sources, all checked 30 Aug 2026
- OpenAI, “Overview of OpenAI Crawlers” (OAI-SearchBot, OAI-AdsBot, GPTBot, ChatGPT-User; allow one and disallow the other; undated): developers.openai.com/api/docs/bots
- Anthropic, “Does Anthropic crawl data from the web?”, stamped 7 Apr 2026 (ClaudeBot, Claude-SearchBot, Claude-User; what disabling each does; per subdomain): support.claude.com/en/articles/8896518
- Perplexity, “Perplexity Crawlers”, stamped 29 Jan 2026 (PerplexityBot; Perplexity-User “generally ignores robots.txt rules”; firewall steps): docs.perplexity.ai/docs/resources/perplexity-crawlers
- Apple, “About Applebot” (Applebot and Applebot-Extended; the Googlebot-instruction fallback; no crawl-delay): support.apple.com/en-us/119829
- Google, “List of Google’s common crawlers” (eleven tokens; the Google-Extended training, grounding and no-impact-on-Search wording): developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- Google, “List of Google user-triggered fetchers” (nine fetchers; Google-Agent; Google-NotebookLM replaced by Google-GeminiNotebook; Web Bot Auth): developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
- Google Search Central, “AI features and your website”, updated 10 Dec 2025 (Googlebot as the control for AI features in Search; snippet controls): developers.google.com/search/docs/appearance/ai-features
- Google Search Central, “Google’s Guide to Optimizing for Generative AI Features on Google Search”, updated 10 Jul 2026 (generative features “rooted in our core Search ranking and quality systems”): developers.google.com/search/docs/fundamentals/ai-optimization-guide
FAQ
Frequently asked questions.
Is blocking Google-Extended still free?
Not entirely, and that changed. Google’s crawler documentation now describes Google-Extended as controlling grounding as well as training, meaning Search-index content served to the model at prompt time in Gemini Apps and on Vertex AI. The same entry still states it does not affect inclusion in Google Search or act as a ranking signal there. Disallowing it costs you the Gemini assistant while leaving AI Overviews and AI Mode alone.
Can I stop AI assistants from fetching my pages when a user asks?
Usually not with robots.txt. OpenAI writes that ChatGPT-User rules may not apply because the action is user-initiated, Perplexity writes that Perplexity-User generally ignores robots.txt, and Google says the same of its whole user-triggered fetcher class, Google-Agent included. Anthropic is the exception: Claude-User is a robot whose access you set, and disabling it stops retrieval in response to a user query. Anything stronger needs authentication rather than a directive.
If I only check one line of my robots.txt, which line should it be?
The wildcard group. A broad disallow under User-agent asterisk removes every search-side crawler at once, and it is usually inherited from a staging file or a platform default rather than chosen. Check it before you check any named token, then confirm the five search-side tokens are allowed on every host you own, not just on the main marketing domain. Robots.txt is scoped per host, so subdomains need their own file.
Does robots.txt keep my content out of AI answers?
It keeps the compliant crawlers out, which is not the same thing. A directive is a request rather than an access control, the user-triggered fetchers largely ignore it by design, and nothing in your file governs what other people’s pages say about you. An engine can describe your product accurately from a review, a forum thread or a competitor comparison it retrieved somewhere else entirely.
Bavior Editorial
The team that researches and maintains Bavior’s writing on Reddit marketing and AI search visibility. Every figure here is attributed to a named source with the date it was checked, and none of our links are affiliate links.
Found a number that looks wrong? Tell us and we will re-check it: support@bavior.com