Technical GEO · Scale

What does real AI crawl traffic look like?

Not how to read your logs, but what the traffic in them actually is: how much of it there is next to classic search crawling, which part of it grew, what it fetches, and which capacity decisions that supports.

Topic in Technical GEO12 min readUpdated 30 Aug 20267 sources, all primary

Back to the roadmap Next: Paywalls and crawler blocks

On this page

The short answer

Smaller than the coverage implies, and growing in the part of it that changes no answer. Across one large network in 2025, search engine crawlers drove 40% of verified bot traffic and AI crawlers half of that, 20%, and search engine optimization bots over 13%.¹ The useful move is to stop reading AI crawling as one line and start reading it as three series that went in three different directions.

Key takeaways

Crawling for model training dominates the volume, reaching as much as 7 to 8 times search crawling at peak, while the search-side crawl that actually feeds citations finished 2025 just under 10% below where it started.
Not all of it is your HTML. On one measured site, one assistant’s fetches were 57.70% HTML while another’s were 35.17% images, which makes the bandwidth complaint a different complaint per vendor.
The network average is a weak prior for your site: retail and computer software alone drew just over 40% of AI crawler activity in October 2025.
The aggregate share itself swung from 2.4% to 6.4% inside one year, so a swing of that size in your own logs may be the market moving rather than anything you shipped.

Absolute size

How big is AI crawling next to classic search crawling?

Roughly half the size, on the one network large enough to publish the comparison, and concentrated in a handful of user agents rather than spread across the field.

Cloudflare’s 2025 year in review, covering 1 January to 2 December 2025 across its whole network, put search engine crawlers at 40% of verified bot traffic for the year and AI crawlers at half of that, 20%.¹ Inside those categories the concentration is extreme rather than long-tailed: Googlebot alone drove over 28% of verified bot traffic, OpenAI’s GPTBot about 7.5%, and Microsoft’s Bingbot 6%.¹ Four user agents carry most of what gets described as an army of crawlers.

Restricting the count to HTML, which is the traffic carrying your actual content, tightens the picture again. Across 2025 AI bots other than Googlebot averaged 4.2% of HTML requests on that network while Googlebot on its own accounted for 4.5%, and as of 2 December humans generated 47% of HTML requests against 44% for non-AI bots.¹ More than nine HTML requests in ten had nothing to do with an AI crawler at all. A hosting platform’s own logs from a month in late 2024 reached the same shape by a different route, putting the two largest AI crawlers together at about 20% of Googlebot’s fetch volume in the same period.³

One caveat travels with every number here. Googlebot and Bingbot crawl for search indexing and for AI training at the same time, so whether they sit in the search bucket or the AI bucket is a labelling decision rather than a fact about the packets. Cloudflare counts Googlebot in both, which is why its AI crawler overview is headed by a bot most people would not call an AI crawler.¹

Growth

Which part of AI crawling actually grew?

Cloudflare classifies AI crawling by declared purpose: training, which gathers content to build a model; search, which builds the index an answer is grounded in and may include retrieval-augmented generation; and user action, which fires because a person just asked an assistant something. A fourth bucket holds crawlers whose purpose is undeclared or unclear.¹ The three declared classes moved in three different directions during 2025, and the divergence is the whole finding.

Training dominates the volume, reaching as much as 7 to 8 times search crawling and 32 times user action crawling at peak.¹ User action crawling grew fastest: it began the year as the smallest of the three, more than doubled through January and February, doubled again in early March and kept climbing. Cloudflare’s own heading for that series says it increased by over 15x in 2025, while the paragraph beneath it says up over 21x from January through early December.¹ The post does not reconcile the two, so the conservative figure is the one worth quoting and the gap is worth knowing about.

Search crawling is the series nobody puts in a headline, and it is the one a citation depends on. It was strongest through mid-March, dropped by approximately 40%, recovered gradually, and ended the surveyed period just under 10% below where it started the year.¹ Individual crawlers diverged as sharply: OAI-SearchBot spiked in late October to roughly 5 times its January volume, PerplexityBot finished about 3.5 times higher, ClaudeBot effectively doubled through the first half then fell back to around 10% above its start, and GPTBot peaked in June and ended November only slightly above where it began.¹ Which token belongs to which purpose is documented by each vendor and tabulated in the log-reading article.⁴⁵⁶

Composition

How much of an AI crawler’s traffic is your HTML?

Not all of it, and the split differs enough between vendors to change what blocking any one of them would save. Vercel and MERJ measured the content types AI crawlers fetched on nextjs.org and found ChatGPT’s fetches were 57.70% HTML while Claude’s were 35.17% images, with both spending a further slice on JavaScript files they do not execute, 11.50% and 23.84% respectively.³ Googlebot’s fetches across Gemini and Search were spread differently again: 31.00% HTML, 29.34% JSON, 20.77% plain text and 15.25% JavaScript.³

Two things follow for capacity. The complaint that AI crawlers are eating your bandwidth is a different sentence for each vendor, because one of them is mostly pulling documents from your origin and another is mostly pulling images, and those land on different lines of your bill. And a crawler fetching script bundles it never runs is buying bytes it cannot read, which makes that portion an asset and caching problem rather than a content problem. Whether any of them execute JavaScript at all is its own question.

The sample deserves stating plainly, because content-type mix is exactly the kind of statistic that travels badly. Those percentages come from one documentation-heavy site on one framework in one month at the end of 2024, and no replication has been published in the twenty months since. A media site heavy with imagery, or a store with a large product catalogue, will not see that distribution. Pull the same breakdown from your own edge logs before planning around anybody else’s.

Cost

What does this traffic actually cost you?

Less than the argument about it usually assumes, and one query settles it faster than a meeting. At the network average, AI bots other than Googlebot account for about one HTML request in twenty-four.¹ On a site serving a hundred thousand HTML requests a month that is roughly four thousand fetches, which is not a capacity event on any modern stack. A team weighing whether to block would do better to group one month of edge logs by user agent class, status and bytes served first, because the number either is or is not large enough to justify the rest of the conversation.

Where the cost does bite is narrower than the word traffic suggests. It concentrates in HTML that misses the cache and reaches your application, in image egress from the crawler that mostly wants images, and in error pages your framework renders dynamically instead of returning from the edge. Each of those is a caching and asset decision, each one helps human visitors as well, and each is cheaper than a policy fight.

The network average is also a poor prior for any particular site. In October 2025 retail and computer software together attracted just over 40% of all AI crawler activity, and the top ten industries just under 70%.¹ By region, Googlebot took between 35% and 55% of crawler traffic everywhere Cloudflare measured, with GPTBot or Bingbot second at 13% to 14%.¹ A software documentation site and a local services site are not drawn from the same distribution, and neither of them is the average.

Baselines

When has your own crawl traffic really changed?

Less often than a chart suggests, because the baseline itself moves. The aggregate AI share of HTML requests across a network of millions of sites ranged from 2.4% in early April to 6.4% in late June inside the same year, a swing of more than two and a half times with no site doing anything.¹ A doubling in your own crawl series over a quarter is therefore consistent with the market moving underneath you, and attributing it to something you shipped needs the comparison rather than just the series. The live Radar series that review is drawn from is public, so making the comparison costs nothing.²

Two further reading errors are common and cheap to avoid. A single aggregated line for AI bots hides the divergence above, where three named crawlers finished the same year at roughly 3.5 times, 1.1 times and flat. And an aggregate that includes dual-purpose Googlebot is a different series from one that excludes it, by roughly its own size again, so two charts that disagree may simply be counting different populations.¹

Whether a movement in your own counts is real at your volume is a separate statistical question with a separate answer, worked through in the article on reading crawler logs. The question here comes before it: a change can be perfectly real on your site and still not be about your site.

Decisions

What should this change, and what should it not?

Three decisions get better with this data and one gets worse. It should change how you budget: cache HTML at the edge, serve right-sized images, return errors from the edge rather than the application, and the crawl is absorbed without anyone touching a robots.txt file. It should sharpen how you read your own token policy, because a training crawl costs you bytes and returns nothing on the answer side while the search-side crawl is the one a citation depends on. And it should reset expectations, because a crawl spike is an input, not a result.

What it should not change is anything you claim about visibility. No published work links per-site crawl volume to per-site citation rates, the two datasets share no identifier, and crawl growth is therefore not evidence that anything downstream improved. Whether to gate or block the traffic is a separate decision resting on separate evidence, taken up in the article on paywalls and crawler blocks.

The honest limit of this article

Nearly every figure here comes from two vendors reporting their own networks, one a content delivery network and one a hosting platform, and neither sample is the web. The content-type percentages are from a single site at the end of 2024 and have not been replicated. The 2025 shares stop on 2 December 2025, and the crawler mix has moved since. None of it has been validated against citation outcomes, because no published study connects per-site crawl behaviour to per-site citation rates. Read all of it as a plausibility range and your own edge logs as the measurement.

Where a product fits, and where it does not

Everything on this page is a query against logs you already own: group a month of edge requests by user agent class, status and bytes, and you have the shape of your own AI crawl traffic without buying anything. Bavior works on the answer side instead, running a fixed prompt panel across five engines on a schedule and recording which sources each answer cited. What Bavior does not do is read your access logs, count crawler bytes, size your cache, or tell you that a crawl increase caused a citation, and nobody can do that last one, because the crawl side and the answer side share no identifier. The free visibility check and the free GEO audit run without a paid plan; paid plans are from $99/mo billed monthly, or $79.17/mo billed annually (as of 30 Aug 2026).

Sources, all checked 30 Aug 2026

Cloudflare Radar 2025 Year in Review, data 1 Jan to 2 Dec 2025; verified bot category shares, per-bot shares, purpose classes and their trends, HTML request shares, robots.txt dispositions, October 2025 crawling by region and industry: blog.cloudflare.com/radar-2025-year-in-review
Cloudflare Radar, AI Insights (the live series behind the year in review, including bot best practices and AI user agents found in robots.txt): radar.cloudflare.com/ai-insights
Zecchini, Moore, Ubl, Siddle, “The rise of the AI crawler”, Vercel with MERJ, 17 Dec 2024; first-party log data from nextjs.org and the Vercel network for the sampled month; content-type shares per crawler and relative fetch volumes: vercel.com/blog/the-rise-of-the-ai-crawler
OpenAI, crawler and bot documentation (GPTBot for training, OAI-SearchBot for search, ChatGPT-User for user-triggered fetches; first-party): developers.openai.com/api/docs/bots
Anthropic, “Does Anthropic crawl data from the web?” (ClaudeBot, Claude-SearchBot, Claude-User; first-party): support.claude.com/en/articles/8896518
Google Search Central, “Google user-triggered fetchers” (the user-triggered class as its own documented category; first-party): developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
Perplexity, bots documentation (PerplexityBot and Perplexity-User; first-party): docs.perplexity.ai/guides/bots

Continue reading

Learn GEO GEO roadmap Learn GEO Previous: Do AI crawlers run JavaScript Learn GEO Next: Paywalls and crawler blocks

FAQ

Frequently asked questions.

Is AI crawler traffic bigger than Googlebot traffic yet?

No, not on the one network large enough to publish the comparison. Across 2025 Cloudflare put search engine crawlers at 40% of verified bot traffic and AI crawlers at half of that, 20%. Counting HTML requests only, AI bots other than Googlebot averaged 4.2% of them while Googlebot alone accounted for 4.5%. AI crawling is real and growing, but it is roughly the size of Googlebot rather than a multiple of it.

Which kind of AI crawling grew in 2025?

Training and user-action crawling grew; search crawling did not. Cloudflare reported training as the overwhelming majority of AI crawler volume, reaching 7 to 8 times search crawling at peak, and user-action crawling up over 15x across the year. Search crawling, the class that builds the index an answer is grounded in, ended the surveyed period just under 10% below where it started. The headline growth is not in the class that produces citations.

Do AI crawlers only fetch HTML pages?

No, and the mix differs sharply by vendor. On one measured site, ChatGPT's fetches were 57.70% HTML while Claude's were 35.17% images, and both spent a further 11.50% and 23.84% on JavaScript files they do not execute. Googlebot's fetches across Gemini and Search were more evenly spread at 31.00% HTML, 29.34% JSON, 20.77% plain text and 15.25% JavaScript. Bandwidth complaints therefore mean different things per crawler.

Is AI crawl traffic expensive enough to justify blocking it?

Usually not on volume alone, and the check is one query. At the network average, AI bots other than Googlebot are about one HTML request in twenty-four, so a site serving a hundred thousand HTML requests a month sees roughly four thousand AI fetches. Group a month of your own edge logs by user agent class and bytes served before deciding. Caching, right-sized images and edge-served errors usually settle it.

My AI crawl volume doubled this quarter. Did something change on my site?

Possibly not, because the baseline moves on its own. The aggregate AI share of HTML requests across a network of millions of sites ranged from 2.4% to 6.4% inside a single year, a swing of more than two and a half times that no individual site caused. Compare your series against a public network series before attributing the change to anything you shipped, and split it per crawler rather than reading one aggregated line.

Bavior Editorial

The team that researches and maintains Bavior’s writing on Reddit marketing and AI search visibility. Every figure here is attributed to a named source with the date it was checked, and none of our links are affiliate links.

Found a number that looks wrong? Tell us and we will re-check it: support@bavior.com