Why do crawlability and indexation decide whether you can be cited?

Google requires exactly one thing, and it is an indexing requirement: “a page must be indexed and eligible to be shown in Google Search with a snippet, fulfilling the Search technical requirements.” The same documentation set says the generative features “are rooted in our core Search ranking and quality systems”, that “there are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary”, and that “you don't need to create new machine readable files, AI text files, or markup to appear in these features. There's also no special schema.org structured data that you need to add.”¹

Read that as a boundary rather than encouragement. It rules out an entire category of paid advice: AI-specific markup, AI-specific files, AI-specific submission. The burden goes back to the ordinary work of being indexable, fetchable and snippet-eligible. The same docs state that Google Search ignores llms.txt, retiring the most popular AI-file proposal of the last two years on the authority of the engine it was meant to influence.

Two Search-side controls do work and are worth knowing precisely: nosnippet, max-snippet and data-nosnippet limit what Google may show from your page, and noindex removes it entirely. Blocking a training crawler is not on that list, for reasons the fourth section covers.

Often enough to be the first thing you check: a 2026 preprint, reported via the critical survey, found 27.1% of URLs already cited in AI answers were not scraped because they were inaccessible, removed or non-textual, and separately labelled about 16% of the 19,154 pages it did retrieve as AI-generated.² The denominator is worth naming: it measures whether a third-party researcher could fetch a cited URL, not whether the engine that cited it could. The failure modes are mundane: a login wall, a consent interstitial, a bot-blocking WAF rule, an image-only page, a URL that has since 404ed. Each is a technical ticket, not a content decision.

Two beliefs about access are wrong in the other direction. The first is that a paywall keeps you out of AI answers: a vendor study of 100,000 long-tail US news queries collected in April 2025 found over 96% of one major newspaper's AI Overview citations pointing at paywalled content, because Google's flexible sampling lets Googlebot behind the paywall. Vendor-published; described rather than linked, per this curriculum's sourcing rule.

The second is that blocking a crawler protects you from being summarised. Tow Center at Columbia tested this in October 2025 and found that agentic browsers retrieved a 9,000-word subscriber-only article that the standard interfaces of the same two products could not, because the publisher had blocked those crawlers but the browsers are indistinguishable from Chrome in server logs. Where the block genuinely held, the agent “produced a composite summary, drawing on tweets about the article, syndicated versions, citations in other outlets, and related coverage across the Web”.⁶ Blocking changes who gets the traffic; it does not remove you from the answer.

robots.txt is a request; your server is the enforcement, and the two disagree more often than anyone expects. A bot-mitigation rule, a rate limiter or a geographic block can drop a crawler that robots.txt explicitly allows, and nothing reports it, because a fetch that never completed leaves no impression to be missing. That gap belongs to the same family as the 27.1% figure above, and unlike the engines' internal behaviour it is entirely yours to inspect.

Check the access log rather than the config file, and check by address rather than by name. Every vendor in the table above publishes machine-readable IP ranges for exactly this purpose. OpenAI lists one JSON file per token, for OAI-SearchBot, GPTBot and ChatGPT-User.³ Perplexity publishes one for PerplexityBot and one for Perplexity-User, and points firewall operators at those endpoints as the source of truth.⁵ Google publishes common-crawlers.json alongside separate files for special-case crawlers and user-triggered fetchers, and documents a manual alternative: a reverse DNS lookup on the fetching address that must resolve to googlebot.com, google.com or googleusercontent.com, then a forward lookup back to the same address.¹⁰

Two consequences are worth acting on. A user-agent string is not evidence, so allowing a bot by name admits anything willing to send that name, and blocking by name stops nothing that matters. And if a search token shows no successful fetches across a week while the pages are live and allowed, the block sits between robots.txt and your origin, which makes it a firewall ticket rather than an SEO one.