Do paywalls and crawler blocks keep you out of AI answers?

It asks a fetcher not to request a path. Documented, well-behaved crawlers honour that request, and it has never been an access control: the file is public and enforcement happens entirely on the other side. The infrastructure provider behind the most widely deployed managed version of that file says as much about its own newer signalling scheme: content signals “express preferences; they are not technical countermeasures against scraping. Some companies might simply ignore them.”⁴

The split that decides most outcomes is not training against search, which the retrieval stage covers. It is scheduled crawling against a fetch that happened because a person asked for something, and on that boundary the vendors contradict each other in their own documentation. Every quotation below was checked against the live vendor page on 30 August 2026.

Read the last row twice. An agentic browser is a browser a person is driving, not a crawler with a polite user agent: no directive addresses it, and blocking its traffic pattern means blocking Chrome.

That depends on which kind you built, and many publishers built the kind that does not gate a machine. A client-side overlay sends the full article to the browser and hides it behind a subscribe prompt. The Tow Center named MIT Technology Review, National Geographic and the Philadelphia Inquirer as using it, and observed that “while this content is invisible to humans, AI agents like Atlas and Comet can still read it.” A server-side paywall, which it attributed to the Wall Street Journal and Bloomberg, sends no text until credentials are verified.² Neither stops a subscribed reader pointing an agent at the page.

Its October 2025 demonstration is the cleanest published case of the two failure directions meeting. Asked for the full text of a nine-thousand-word subscriber-exclusive article in MIT Technology Review, two agentic browsers returned it; the same prompt in the standard chat interfaces of the same two companies was refused, because the Review had blocked their crawlers. The block worked on the crawler and was irrelevant to the browser.

The indexing half has a documented answer that deceives nobody. Google’s structured-data guidance for subscription and paywalled content exists, in its own words, because the markup “helps Google differentiate paywalled content from the practice of cloaking, which violates spam policies”, and it opens by narrowing its scope: “This guide only applies to content that you want crawled and indexed.”⁹ Marking the gated section with isAccessibleForFree and hasPart is how a page stays eligible while you still charge for it. Serving a crawler text you hide from the reader without that markup is what the policy calls cloaking, and this curriculum will not hand you that.

Three couplings turn a narrow intention into a wide loss. At Google there is no AI-only directive: “AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search.” The finer controls it names, nosnippet, data-nosnippet, max-snippet and noindex, all cut your ordinary result too, and Google-Extended governs training and grounding in other Google systems, not Search.⁸ Of the tools with public crawlers in the Tow Center’s sample, one was blocked by none of the twenty publishers, which the researchers put down to it sharing a crawler with a general search engine: blocking it meant leaving that index.¹

The third inverts the effect you wanted. Anthropic documents that “alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file.”⁶ A firewall rule aimed at a crawler can leave you both fetched and unheard, because the file carrying your preference is the thing you blocked.

Timing quietly ruins most before-and-after readings. OpenAI states it “can take ~24 hours from a site’s robots.txt update for our systems to adjust”; Perplexity says changes may take up to 24 hours; Google says preview controls “can take anywhere from several days to several months”.⁵⁷⁸ A block is not a switch, and neither is unblocking.