Home/Learn GEO/Reading crawler logs
Measuring AI visibility · The crawler side

How do you read AI crawler logs for GEO?

Answer-side sampling tells you the outcome; server logs tell you whether the input ever arrived. The whole analysis is one filter, three questions and one verification step, in that order, because the order is what stops you reading noise.

Topic in Measuring AI visibility12 min readUpdated 30 Aug 202611 sources, all primary
Back to the roadmap Next: What engines quote
On this page
Share this
Share on X Share on LinkedIn
The short answer

Filter your access logs by user-agent token, keep the search-side, training-side and user-triggered classes apart, and never aggregate them into one line. Then read three things in that order: the error rate per token, whether your highest-intent pages are fetched at all, and where the user-triggered fetchers land. Start with errors because they are usually the largest single loss: in one infrastructure provider’s December 2024 log study, GPTBot spent 34.82% of its fetches on 404s against Googlebot’s 8.22%.6

Key takeaways
  • Three classes, not two: search-side tokens (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot, Googlebot), training-side tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) and user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User, Google-Agent). Every split is documented by the vendor itself.
  • Fix wasted fetches first: sorting your own 404s by fetch count gives a ranked list of dead internal links and stale sitemap entries.
  • A zero beats a percentage. A high-intent page with no successful search-side fetch in 30 days is a real finding; a count that fell from 12 to 7 is noise.
  • A user-agent header is self-declared, so verify it against the vendor’s published address ranges before you trust any of it.
The filter

Which user-agent tokens should you count separately?

Count three classes separately, not two: the search-side crawler that builds the index an answer is grounded in, the training-side crawler that changes no answer, and the user-triggered fetcher that arrives when a person asks.

Every major vendor documents the split itself, so the table below is first-party rather than inferred. The third column is the one most log reports omit.

VendorSearch-side tokenTraining-side tokenUser-triggered fetcher
OpenAIOAI-SearchBotGPTBotChatGPT-User1
AnthropicClaude-SearchBotClaudeBotClaude-User2
PerplexityPerplexityBotNone documentedPerplexity-User3
AppleApplebotApplebot-ExtendedNone documented4
GoogleGooglebotGoogle-Extended5Google-Agent10

Aggregating the three classes into one “AI bots” line is the mistake that makes a log report useless, because the three answer different questions. A drop in search-side fetches is a retrieval problem you should fix this week. A drop in training-side fetches means nothing for any answer. And a rise in user-triggered fetches is not a maintenance signal at all: it is somebody asking about you right now, which is the closest thing to demand data this field offers.

Check the table against vendor documentation rather than against a blog post, this one included, because the list moves. OpenAI now documents a fourth token, OAI-AdsBot, which validates pages submitted as ads;1 Google documents user-triggered fetchers as a class of their own, including Google-Agent, and lists the former Google-NotebookLM as supported only until August 2026, with Google-GeminiNotebook taking its place.10 Which token to allow, and what blocking each does, is the technical GEO stage’s subject; this page assumes they are allowed and asks what the traffic tells you.

The three questions

What are the three things to look for, in order?

Look for wasted fetches first, missing coverage second and demand third, because that is the order in which they are fixable. Each is one aggregation over the same filtered log, and none needs a tool beyond whatever already queries your access logs.

First: the status-code mix per token. Group by token and status class, and read the share of 2xx, 3xx and 4xx separately for each. AI crawlers follow site structure markedly worse than Googlebot does. An infrastructure provider’s December 2024 log study, covering 569 million GPTBot and 370 million ClaudeBot fetches in the sampled month, found GPTBot spending 34.82% of its fetches on 404s and 14.36% on redirects, and ClaudeBot 34.16% on 404s, against 8.22% and 1.49% for Googlebot.6 The actionable part is the URL list underneath the percentage: sort your own 404s by fetch count and you get a ranked list of dead internal links and stale sitemap entries.

Second: coverage of your highest-intent pages. Write down the ten to twenty URLs that would answer a buying question, meaning pricing, comparisons, integrations, security and objection pages, and check each against the log for a successful search-side fetch in the last 30 days. A page nobody retrieves cannot be cited, so a zero here is a stronger finding than any rate on the answer side. Two failure shapes recur: pages fetched by one vendor’s search token and no other, which usually points at a robots.txt or rendering difference; and pages fetched once months ago and never revisited, which usually points at weak internal linking to them.

Third: user-triggered arrivals. Count fetches by ChatGPT-User, Claude-User and Perplexity-User separately, by day and by path. These do not run on a crawl schedule; they happen because a person asked a question and the assistant went to look. Read them as a demand signal on the pages that received them, and watch for a new path appearing: a comparison page that starts drawing user-triggered fetches is telling you a shortlist conversation is happening about you. Cloudflare’s network-wide 2025 review reported user action crawling growing more than fifteenfold across the year.7

Calibration

What volume should you expect, and when is a change real?

Expect AI crawlers to be roughly the size of Googlebot on your site rather than a multiple of it, and expect most of your own counts to be too small to trend. Cloudflare’s 2025 year in review, covering 1 January to 2 December 2025 across its network, put AI bots at 4.2% of HTML requests for the year against Googlebot at 4.5%.7 The same December 2024 log study put monthly volumes at 4.5 billion for Googlebot against 569 million for GPTBot, 370 million for ClaudeBot, 314 million for Applebot and 24.4 million for PerplexityBot.6 Scale those ratios to your own Googlebot traffic and you have a rough expectation.

The statistical point matters more than the benchmark, and it is the one log reports get wrong. Fetch counts on a single site are small numbers, and small counts are noisy in a way percentages hide. If a search-side token fetched 12 pages last week and 7 this week, that is not a 42% decline; for counts this small the ordinary Poisson spread alone spans roughly that range, and reading a trend into it is an error. The rule that follows: aggregate to at least a month before comparing counts, and treat a fetch count under about 30 in a period as a presence-or-absence observation rather than a quantity. The article on statistical power works through the same problem on the answer side.

Referrals

What does the crawl-to-referral ratio tell you?

It tells you how much a platform takes from your site against how much it sends back, and both halves already sit in the same access log. Cloudflare computes it by dividing HTML requests from a platform’s crawler user agents by HTML requests whose Referer header names that platform.11 You can compute the same quantity yourself: crawler fetches on one side, and on the other the sessions arriving with a chatgpt.com, perplexity.ai, claude.ai or gemini.google.com referrer.

Network-wide the ratios are lopsided, and they differ by orders of magnitude between platforms. Across 2025 Cloudflare put Anthropic’s in the 25,000:1 to 100,000:1 band after May, with earlier peaks as high as 500,000:1; OpenAI’s spiky, peaking at about 3,700:1 in March; Perplexity’s generally below 400:1; and Google’s at a little over 3:1, briefly reaching 30:1 in April before falling back.7 Two cautions before you read your own. The referral side undercounts, because a browser does not always send a Referer header and some clients strip it, so your measured ratio is an upper bound. And a high ratio is not by itself a verdict: it describes an exchange rate, not whether being crawled was worth it, and on a site whose goal is being cited the click was never the payment being asked for.

Verification

How do you know the bot was really who it said it was?

You do not, until you check the address it came from, because a user-agent string is a self-declared header that anything can send. Verification turns a log line into evidence: match the source address against the vendor’s published crawler ranges, or do a reverse DNS lookup and then a forward lookup on the result. OpenAI publishes a separate address file per token, Anthropic publishes one file covering all three of its crawlers, and Apple and Perplexity publish theirs too,1234 so doing this once as a filter rule is cheaper than doing it as an investigation later.

The reason to bother is that undeclared crawling is a live dispute rather than a hypothetical. Cloudflare published an analysis on 4 August 2025 stating that it registered brand-new, never-indexed domains with a robots.txt disallowing all crawling, and that one assistant still returned detailed content about them; it attributed the traffic to a generic browser user agent from addresses outside the vendor’s published ranges and de-listed that vendor as a verified bot.8 The vendor published a rebuttal the following day denying stealth crawling and attributing the requests to a third-party cloud-browser service. That disagreement has not been resolved publicly. Treat it as a reason to verify rather than as a settled finding about any one company.

Limits

What can server logs not tell you?

Logs cannot tell you whether a fetch turned into a citation, and that is the gap that keeps the answer-side panel necessary. A page can be fetched by a search-side crawler every week and cited by nothing; a page can be cited from an index built months ago with no recent fetch at all. Join the two datasets by time and read them side by side, and resist the causal sentence, because “the crawler came back and then we started appearing” is a sequence, not a mechanism.

Three more blind spots are worth naming. Agentic browsers are indistinguishable from an ordinary browser session in your logs: a Tow Center for Digital Journalism analysis published 30 October 2025 found they retrieved subscriber-only material that the same vendors’ standard interfaces could not, precisely because nothing in the request marked them as automated.9 A CDN or edge cache hides fetches from origin logs entirely, so read at the edge if you can and expect origin-only figures to undercount. And a token that stops appearing is ambiguous: it may have been blocked, renamed, or simply have nothing new to fetch.

What logs do give you is the only first-party evidence in this field. Everything on the answer side is sampled from somebody else’s system, which makes the access log the right place to start an investigation even though it is the wrong place to end one.

The honest limit of this article

The quantitative anchors here are older and narrower than they look. The 34.82% and 34.16% error rates come from one infrastructure provider’s December 2024 log study of its own network, with no replication published in the twenty months since; the crawlers have certainly changed in that window, and your own error rate is the number that governs your site. The Cloudflare Radar shares and crawl-to-refer ratios are network-wide, and a documentation-heavy site and a news site see quite different mixes. Read both as calibration for what is plausible, and treat your own logs as the measurement. Nothing here has been validated against citation outcomes, because no published study links per-site crawl behaviour to per-site citation rates.

Where a product fits, and where it does not

All of this is free and it is yours already: filter the access log by user-agent token, group by status class, check your twenty highest-intent URLs for a successful search-side fetch, count the user-triggered fetchers by day, and verify the addresses against the vendors’ published ranges. Bavior works on the other side of the same question: it runs a fixed prompt panel across five engines on a schedule and records which sources each answer cited, which tells you whether a crawl-side fix changed who shows up. What it does not do is read your server logs, verify crawler addresses, fix your 404s or tell you whether a particular fetch became a particular citation, and nobody can do that last one, because the two datasets share no identifier. The free visibility check and the free GEO audit run without a paid plan; paid plans are from $99/mo billed monthly, or $79.17/mo billed annually (as of 29 Aug 2026).

Sources, all checked 30 Aug 2026
  1. OpenAI, crawler and bot documentation (GPTBot, OAI-SearchBot, OAI-AdsBot, ChatGPT-User, address file per token; first-party): developers.openai.com/api/docs/bots
  2. Anthropic, “Does Anthropic crawl data from the web?” (ClaudeBot, Claude-SearchBot, Claude-User, published address ranges; first-party): support.claude.com/en/articles/8896518
  3. Perplexity, bots documentation (PerplexityBot, Perplexity-User, published address ranges; Perplexity-User “generally ignores robots.txt rules”; first-party): docs.perplexity.ai/guides/bots
  4. Apple, “About Applebot” (Applebot, Applebot-Extended, published CIDR file, browser rendering; first-party): support.apple.com/en-us/119829
  5. Google Search Central, “AI features and your website” (Googlebot, Google-Extended, indexing and snippet eligibility; first-party): developers.google.com/search/docs/appearance/ai-features
  6. Zecchini, Moore, Ubl, Siddle, “The rise of the AI crawler”, Vercel, 17 Dec 2024; first-party log data, 569M GPTBot and 370M ClaudeBot fetches in the sampled month; 404 and redirect shares per token: vercel.com/blog/the-rise-of-the-ai-crawler
  7. Cloudflare Radar 2025 Year in Review, data 1 Jan to 2 Dec 2025; AI bots 4.2% of HTML requests, Googlebot 4.5%, user action crawling up over 15x, crawl-to-refer ratios by platform: blog.cloudflare.com/radar-2025-year-in-review
  8. Cloudflare, “Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives”, 4 Aug 2025 (infrastructure first-party; the vendor published a rebuttal on 5 Aug 2025 denying stealth crawling): blog.cloudflare.com
  9. Tow Center for Digital Journalism, “How AI browsers sneak past blockers and paywalls”, 30 Oct 2025; agentic browsers indistinguishable from an ordinary browser session in logs: cjr.org
  10. Google Search Central, “Google user-triggered fetchers” (Google-Agent, Google-GeminiNotebook, Google-NotebookLM supported until August 2026; these fetchers generally ignore robots.txt; first-party): developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
  11. Cloudflare, “The crawl before the fall… of referrals”, 1 Jul 2025; defines the crawl-to-refer ratio as HTML requests from a platform’s crawler user agents divided by HTML requests whose Referer header names that platform: blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar

Continue reading

Learn GEO GEO roadmap Learn GEO Previous: What cannot be measured Learn GEO Next: What engines quote
FAQ

Frequently asked questions.

Which AI crawler tokens should I look for in my server logs?

Separate three classes rather than two. The search-side tokens build the index an answer is grounded in: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Applebot and Googlebot. These are the ones that matter for citation. The training-side tokens, meaning GPTBot, ClaudeBot and Applebot-Extended, have no effect on any answer. Google-Extended is no longer purely a training token: Google documents it as controlling grounding in Gemini Apps and on Vertex AI as well, so disallowing it removes you from that surface, though not from Google Search or AI Overviews. The user-triggered fetchers, meaning ChatGPT-User, Claude-User, Perplexity-User and Google-Agent, arrive because a person just asked a question, which makes them a demand signal rather than a maintenance one. Every one of those splits is documented by the vendor itself, so this is a first-party taxonomy rather than an inferred one.

My AI crawler fetches dropped from 12 to 7 last week. Should I worry?

No, because counts that small carry more spread than the change you are looking at. At a dozen fetches a week the ordinary random variation alone covers roughly that range, so reading a 42% decline into it is the same error as drawing a week-over-week arrow on a handful of prompt runs. Aggregate to a month before comparing counts, and treat any fetch count under about 30 in a period as a presence-or-absence observation rather than a quantity. What is worth acting on at small volumes is a zero: a high-intent page with no successful search-side fetch in 30 days is a real finding at any site size.

Can I trust the user-agent string in my logs?

Not on its own, because a user-agent is a self-declared header that anything can send. Verify it against the source address: match against the vendor's published crawler address ranges, or run a reverse DNS lookup and then a forward lookup on the result. OpenAI, Anthropic, Perplexity and Apple all publish address files in their bot documentation, and building the check into a filter rule once is far cheaper than running it as an investigation later. This is not hypothetical. Cloudflare published an analysis in August 2025 alleging undeclared crawling from addresses outside one vendor's published ranges, which that vendor denied the following day, and the disagreement has not been resolved publicly.

Do server logs tell me whether a fetch led to a citation?

No, and nothing else does either, because the two datasets share no identifier. A page can be fetched by a search-side crawler every week and cited by nothing, and a page can be cited from an index built months ago with no recent fetch at all. Join the log side and the answer side by time, read them next to each other, and stop short of the causal sentence, because "the crawler came back and then we started appearing" describes a sequence, not a mechanism. The log is the strongest first-party evidence you have about what reached your server, which makes it the right place to start an investigation and the wrong place to end one.

What is a normal crawl-to-referral ratio for an AI platform?

There is no normal, and the spread between platforms is the finding. Across 2025 Cloudflare put Anthropic's network-wide ratio in the 25,000:1 to 100,000:1 band after May, OpenAI's peaking around 3,700:1 in March, Perplexity's generally below 400:1, and Google's at a little over 3:1. Compute your own by dividing crawler fetches per platform by the sessions arriving with that platform's referrer, then read it as an exchange rate rather than a verdict. Your referral side will undercount, because a browser does not always send a Referer header, so treat the number you get as an upper bound.

Bavior Editorial

The team that researches and maintains Bavior’s writing on Reddit marketing and AI search visibility. Every figure here is attributed to a named source with the date it was checked, and none of our links are affiliate links.

Found a number that looks wrong? Tell us and we will re-check it: support@bavior.com

Your logs are the only first-party evidence.
Read them in the right order.

Start free trial