How AI search works · Stage 3

What happens at the retrieval stage of AI search?

Retrieval is where pages leave the competition without ever being judged on quality, and where the strongest evidence in this whole field sits. Everything downstream is capped by it.

Deep dive on stage 312 min readUpdated 30 Aug 202613 sources, all primary

Back to the roadmap Next: Selection and reranking

On this page

The short answer

Retrieval is the stage where each generated search query hits an index or a live fetcher and comes back with candidate documents. It is a gate rather than a ranking factor: a page that is not indexed, not fetchable, or not relevant to any of the queries the engine issued scores zero at every later stage, however well it is written. A 2026 preprint found 27.1% of URLs cited in AI answers could not even be scraped, being inaccessible, removed, or non-textual.⁷

Key takeaways

Retrievability caps everything downstream: a page that is not a candidate has a probability of zero at selection and at synthesis, whatever its prose does.
Pages drop out at six points: the index, the robots.txt token, the renderer, a dead URL, a gate standing in front of the page, and relevance to the sub-queries actually issued.
Training and search crawlers are separate documented tokens at Google, OpenAI, Anthropic and Apple. Blocking the training one usually changes nothing, though Google-Extended now also controls Gemini Apps grounding; a wildcard disallow removes you from all.
Ship the answer text in the initial HTML response: the only published measurement of JavaScript execution by the classic AI crawlers found none, and nobody has replicated it.

Definition

What happens at the retrieval stage?

Retrieval takes each query produced by fan-out and returns a candidate set of documents for it, either from a pre-built index or from a live fetch at question time. There is nothing exotic about the mechanism; this is search, with all the ordinary consequences that word carries. Google says so about its own AI surfaces in as many words: “our generative AI features on Google Search are rooted in our core Search ranking and quality systems”,¹⁰ and to be eligible as a supporting link a page “must be indexed and eligible to be shown in Google Search with a snippet, fulfilling the Search technical requirements.”¹

Two retrieval patterns coexist and they fail differently. Index-based retrieval means the fetch that mattered happened earlier, possibly weeks ago, by a crawler, and your page as the engine holds it is the version that crawler saw. Live retrieval means a fetcher pulls the URL at the moment someone asks, which makes freshness immediate and makes a slow response, a bot challenge or a 403 fatal in a way it never is for an index. Most of the systems in scope run both, under different user-agent tokens. The stage before this one decides how many separate queries arrive here; the how AI search works stage has the overview.

Failure modes

Where do pages silently drop out?

Pages drop out of retrieval at six distinct points: the index, the robots.txt token, the renderer, the URL itself, a gate standing in front of it, and relevance to the sub-queries the engine actually issued.

Every one of those six is invisible in your analytics, because a page that was never retrieved generates no impression to be missing.

Not in the index

Google’s stated requirement is that the page be indexed and snippet-eligible. A noindex, a max-snippet directive or a canonical pointing elsewhere removes the page from consideration before anything else is evaluated.

The wrong bot is blocked

Every major vendor documents separate tokens for training and for search. Blocking the search token removes you from that engine’s answers. Blocking the training token usually does not, with one documented exception: Google-Extended now also controls grounding in Gemini Apps and on Vertex AI, so disallowing it costs you that surface while leaving Google Search and AI Overviews untouched. A blanket wildcard disallow takes out both.

Text only exists after JavaScript

The only published measurement of the classic AI crawlers found they fetch JavaScript files without executing them. Google, Bing and Applebot render; several others apparently do not, so the risk is uneven rather than universal.

The URL is dead or moved

In one infrastructure provider’s December 2024 log study, GPTBot spent 34.82% of its fetches on 404s and 14.36% on redirects, and ClaudeBot 34.16% on 404s, against 8.22% and 1.49% for Googlebot.⁶

A gate stands in front of it

A login wall, a consent interstitial, a bot-mitigation challenge or an aggressive rate limit all end the fetch. So does content that exists only inside an image, a video or a PDF the fetcher will not parse.

Relevant to nothing that was asked

The quiet one. A page can be perfectly indexed and perfectly written and still match none of the queries fan-out actually issued, in which case it is not rejected at retrieval; it is simply never a candidate.

The size of this population is measurable from the other end. A 2026 preprint, reported via the critical survey, found 27.1% of URLs already cited in AI answers could not be scraped, because they were inaccessible, removed or non-textual.⁷ Read the denominator carefully: it measures whether a third-party researcher could fetch a cited URL, not whether the engine that cited it could. Even so, that is the failure rate among pages good enough to be cited; the rate among pages that never made it that far is necessarily higher, and no one has measured it.

The ceiling

Why is retrievability a ceiling and not a ranking factor?

Because the stages multiply: a page that is not in the candidate set has a probability of zero at selection and at synthesis, and no amount of downstream quality multiplies zero into anything. This is why the field’s strongest experimental results are about position and presence rather than prose. A NeurIPS 2025 benchmark of ten conversational-SEO methods across six domains, more than 1.9k queries and 16k documents, found that in the retail domain moving a source to context position one lifted its citation rank by 2.77 places on average against 0.36 for the best content method tested, and that only 3 of 54 cases were significantly positive.⁸

The 2026 critical survey grades the same boundary explicitly: “query–document relevance and context position are major determinants” is rated high confidence, while a white-hat intervention durably improving organic discoverability across engines is rated low.⁹ Read as a ranking of where to spend effort, that is unambiguous: fix the gate before polishing what is behind it.

robots.txt

Which crawler do you actually need to allow?

Every major vendor documents a split between the bot that gathers training data and the bot that builds the search index. Confusing them is the most common self-inflicted retrieval failure there is.

Vendor	Training / grounding token	Search-index token	Does blocking the first remove you from the answer?
Google	Google-Extended	Googlebot	No. AI Overviews draw on the Search index¹
OpenAI	GPTBot	OAI-SearchBot	No. The docs describe allowing one while disallowing the other²
Anthropic	ClaudeBot	Claude-SearchBot	No. Separate documented tokens³
Apple	Applebot-Extended	Applebot	No. Disallowing the first still permits search inclusion⁴
Perplexity	None documented	PerplexityBot	Its docs state the user-triggered fetcher generally ignores robots.txt⁵

Check all four splits separately and verify with a live fetch rather than an assumption, because the two most common errors point in opposite directions. Disallowing a training token while intending to stay out of AI answers achieves nothing; a blanket wildcard disallow written years ago for a different reason quietly removes every AI search bot at once. Note also what does not help: Google’s documentation states that “you don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.”¹⁰

Rendering

Does the engine render your JavaScript?

It depends on the engine, and the honest summary is narrower than what usually gets repeated. Google renders: its documentation says “Google is able to process content within JavaScript as long as it isn’t blocked”, and its AI surfaces run off that same rendered index.¹⁰ Apple renders: “Applebot may render the content of your website within a browser. If javascript, CSS, and other resources are blocked via robots.txt, it may not be able to render the content properly.”⁴ Agentic browsers render, because they are browsers.

For the rest, there is exactly one published measurement. In December 2024 an infrastructure provider analysed its own log data and concluded that none of the major AI crawlers render JavaScript.⁶ No replication has been published since, the execution-detection method was published separately and is one-sided, and no vendor documents rendering either way. The defensible sentence is “the only published measurement, from December 2024, found no execution”, not “AI crawlers cannot render JavaScript in 2026”.

Either way the fix is cheap enough that the argument does not need resolving. If the prose is present in the initial HTML response, the question never arises for any engine.

Ranking

Does ranking still decide what gets retrieved?

Ranking remains the strongest single predictor of being cited, and it has stopped being sufficient. A 4,706-query audit of Google AI Overviews in Findings of ACL 2026, on September 2025 data, found 53% of the domains it consults absent from the organic top 10 and 27% from the top 100.¹¹ A 55,393-query preprint collected between 13 March and 21 April 2026, roughly twelve times the sample and six months later, found 29.8% of AI Overview reference domains absent from the corresponding first page.¹³ Quote that as a range, roughly 30% to 53% with both collection windows named, rather than as a point from either study; and keep the verb straight, because Kirsten measures the domains the system consults, which is not identical to the domains it cites.

An 11,500-query SIGIR 2026 study measured URL-level Jaccard similarity of 0.11–0.18 between Google organic, AI Overviews and Gemini.¹² Fan-out is the mechanism: retrieval runs per sub-query, so pages that rank nowhere for the visible question can rank precisely for one of its parts.

Be sceptical of any single overlap percentage, including the ones above, because the number depends entirely on a denominator that is rarely stated. “What share of AI answers contain at least one top-ten URL” and “what share of individual citations come from the top ten” are different questions, and published figures across 2025 and 2026 range from roughly a sixth to over three quarters depending on which was asked, when, and on what sample. A number without its denominator and its date is not a measurement. The defensible synthesis is directional: top-ten ranking earns the high-frequency citations, and a large and growing share of the long tail comes from pages that do not rank on page one at all.

The honest limit of this article

The two most-quoted numbers here are the weakest links. The 27.1% unscrapable figure is from a preprint and measures URLs already cited, not the general web. The JavaScript conclusion rests on a single log study from December 2024 whose execution-detection method was published separately and is one-sided: a missing render-time beacon cannot separate a crawler that does not render from one whose beacon was blocked. It is unreplicated twenty months on and no vendor documentation either way, and the products have changed a lot since. Neither claim is settled. The parts of this article that are solid are the first-party robots.txt token splits and Google’s own eligibility statement, because those come from the companies that operate the systems.

Where a product fits, and where it does not

The entire retrieval audit is free and takes about an hour: fetch each important URL with a plain command-line request and confirm the answer text is in the HTML with no JavaScript; check your robots.txt against all four token splits above; confirm the pages are indexed and snippet-eligible; and crawl your own site for 404s and redirect chains, which AI crawlers follow far worse than Googlebot does. No tool is needed for any of that, and no tool can do the fixing. Bavior works on the stage after this one: it runs a fixed prompt set across five engines on a schedule and records which sources each answer cited, which tells you whether a retrieval fix changed who shows up, and where a cited source is a live discussion thread it drafts a reply on an account you control, which you approve before anything posts. It does not crawl your site, does not edit your robots.txt, and cannot make a blocked page retrievable. The free AI visibility check and the free GEO audit run without a paid plan; paid plans are from $99/mo billed monthly, or $79.17/mo billed annually (as of 29 Aug 2026).

Sources, all checked 29 Aug 2026

Google Search Central, “AI features and your website”, updated 10 Dec 2025 (eligibility, Google-Extended, snippet controls; first-party): developers.google.com/search/docs/appearance/ai-features
OpenAI, crawler and bot documentation (GPTBot, OAI-SearchBot, ChatGPT-User; first-party): developers.openai.com/api/docs/bots
Anthropic, “Does Anthropic crawl data from the web?” (ClaudeBot, Claude-SearchBot, Claude-User; first-party): support.claude.com/en/articles/8896518
Apple, “About Applebot” (Applebot, Applebot-Extended, browser rendering; first-party): support.apple.com/en-us/119829
Perplexity, bots documentation (PerplexityBot, Perplexity-User; first-party): docs.perplexity.ai/guides/bots
Zecchini, Moore, Ubl, Siddle, “The rise of the AI crawler”, Vercel, 17 Dec 2024 (infrastructure first-party log data): vercel.com/blog/the-rise-of-the-ai-crawler
Allaham & Diakopoulos, 2026; 27.1% of URLs cited in AI answers could not be scraped (preprint, reported in the critical survey at note 9)
Puerto, Gubri, Green, Oh, Yun, “C-SEO Bench: Does Conversational SEO Work?”, NeurIPS 2025 Datasets & Benchmarks Track, arXiv:2506.11097: arxiv.org/abs/2506.11097
“Optimizing Visibility in Generative Engines: A Critical Survey of Generative Engine Optimization (2023–2026)”, 15 Jul 2026, arXiv:2607.14035 (survey preprint): arxiv.org/abs/2607.14035
Google Search Central, “Google’s Guide to Optimizing for Generative AI Features on Google Search”, updated 10 Jul 2026 (the “rooted in” sentence, JavaScript processing, no AI text files; first-party): developers.google.com/search/docs/fundamentals/ai-optimization-guide
Kirsten et al., Findings of ACL 2026; 4,706-query audit of Google AI Overviews, data collected September 2025 in the US and Germany; “on average 53% (27%) of domains that AIO consults are not contained in top-10 (top-100) Organic search results”: aclanthology.org/2026.findings-acl.526
Grossman et al., SIGIR 2026; 11,500 queries; URL-level Jaccard 0.11–0.18 across Google organic, AI Overviews and Gemini: arxiv.org/abs/2604.27790
Xu, Iqbal & Montgomery, 2026; 55,393 trending queries collected 13 March – 21 April 2026; “29.8% of AIO reference domains do not appear anywhere on the corresponding first page” (preprint): arxiv.org/abs/2605.14021

Continue reading

Learn GEO GEO roadmap Learn GEO Previous: Query fan-out Learn GEO Next: Selection and reranking

FAQ

Frequently asked questions.

If I block GPTBot, will I disappear from ChatGPT's answers?

No. GPTBot and OAI-SearchBot are separate tokens documented separately, and the search-index crawler is the one that governs appearance in search-grounded answers. OpenAI's own bot documentation describes allowing OAI-SearchBot to appear in search results while disallowing GPTBot. The same split exists at Google (Google-Extended versus Googlebot), Anthropic (ClaudeBot versus Claude-SearchBot) and Apple (Applebot-Extended versus Applebot). The rule that actually removes you from every AI answer at once is a blanket wildcard disallow, which is usually inherited from an old robots.txt rather than chosen deliberately.

Does my site need server-side rendering to appear in AI answers?

It depends on the engine, and shipping the text in the initial HTML removes the question for all of them. Google states it "is able to process content within JavaScript as long as it isn't blocked", and Apple documents that Applebot renders within a browser; agentic browsers render because they are browsers. For the classic AI crawlers, the only published measurement, an infrastructure provider's log study from December 2024 found they fetch JavaScript files without executing them; its 569 million GPTBot figure is monthly request volume, not the rendering test's sample, which the post never gives.

Do I still need to rank in Google to be cited by AI search?

Ranking is still the strongest single predictor of being cited, and it is no longer sufficient on its own. A 4,706-query audit of Google AI Overviews published in Findings of ACL 2026 found that on average 53% of the domains an AI Overview consults do not appear in the organic top 10 for the query, and 27% are absent from the top 100. Query fan-out is the mechanism: retrieval runs per sub-query, so a page can be retrieved for part of a question while ranking nowhere for the whole of it. Treat top-ten ranking as what earns the high-frequency citations, and sub-question coverage as what earns the long tail.

Why would a page that is indexed and well written still never get cited?

Most often because it matched none of the queries the engine actually issued. Retrieval runs per fan-out sub-query, not per original question, so a page can be indexed, fast, well written and entirely relevant to the topic while still being a candidate for nothing that was searched. The other common cause is a fetch failure rather than a quality judgement: a 2026 preprint found 27.1% of URLs that AI answers had already cited could not be scraped at all, being inaccessible, removed or non-textual. Neither failure produces any signal in your analytics, which is why they persist.

Bavior Editorial

The team that researches and maintains Bavior’s writing on Reddit marketing and AI search visibility. Every figure here is attributed to a named source with the date it was checked, and none of our links are affiliate links.

Found a number that looks wrong? Tell us and we will re-check it: support@bavior.com