We Analyzed 100,351 AI Bot Visits — Which Crawlers Actually Use Your Content?

For 35 days we logged every request an AI crawler made to quickseo.ai — 100,351 of them, from 30 different bot identities run by 17 companies. Then we did the thing most crawler reports skip: we checked whether each crawler was who it claimed to be, worked out which pages it wanted, and put the requests next to the human visits those same companies sent back.

The headline is not what we expected. One crawler — Meta's meta-externalagent, a training bot — made 64% of all requests, and 51,000 of those landed in an eight-day burst in which it fetched our pricing page roughly every three seconds. The crawler that actually signals a person asking a question right now, OpenAI's ChatGPT-User, was 14%. Anthropic's Claude-User turned out to be mostly Claude Code sessions running on developers' laptops. And Perplexity, for all its 2,390 indexing requests, never once verifiably opened a page live for a user.

This is a one-site case study, not an industry benchmark. quickseo.ai is a small B2B SaaS with a 104-post blog, around 130 free tools and a set of comparison pages, and the mix on a news site or a store will differ. What a single site can give you that a network-wide report can't is the full chain — request, verification, page, and referral — for the same URLs in the same window. Here's what that chain looked like.

How we logged 100,351 AI-bot requests (and what this dataset can't tell you)

The data comes from QuickSEO's own AI Traffic tracker: a server-side hook in the site's Next.js middleware that records a request whenever the User-Agent header contains one of roughly 50 known AI-crawler tokens. Each hit is stored with the crawler's token, the company behind it, the path, the timestamp, the source IP and the full user-agent string, and it's classified into one of four purposes:

• Training — crawlers collecting data to train models (GPTBot, ClaudeBot, meta-externalagent, Bytespider, CCBot, Amazonbot).

• Indexing — search and answer-engine index crawls that decide what's eligible to be cited (OAI-SearchBot, PerplexityBot, Bingbot, Googlebot, Claude-SearchBot).

• AI Answers — a live fetch made because a person just asked something (ChatGPT-User, Claude-User, Perplexity-User, Google-NotebookLM, DuckAssistBot).

• Other — AI-adjacent crawlers that fit none of the above (Diffbot, GrokBot).

Those purposes aren't our invention. Each vendor documents them: OpenAI says ChatGPT-User is "used for certain user actions in ChatGPT and Custom GPTs" and "is not used for crawling the web in an automatic fashion", while OAI-SearchBot is what decides whether a site appears in ChatGPT search results, per OpenAI's bot documentation. Anthropic, Perplexity, Google and Meta all publish equivalent splits. If you want the mechanics of how these bots fetch and render pages — most don't execute JavaScript, which is why a browser-side analytics script never sees them — we've covered how AI crawlers actually work separately. This post is about what they did.

The window is the 35 complete UTC days from 2026-08-16 to 2026-09-19. Four honest limits:

1. No status codes. The tracker records the request before the response exists, so we can't report 404 rates the way Vercel's 2024 study did.

2. Classification is by user-agent string. Anyone can send a request that says Googlebot. So we verified every request we could against the vendors' published IP ranges — that's its own section below.

3. The registry has gaps. Meta's newer meta-webindexer (the crawler it says feeds citations in Meta AI), Google's renamed Google-GeminiNotebook and Google-Agent tokens weren't in the list during this window, so they're invisible here.

4. It's one site. We'll put every number next to a network-wide benchmark where one exists.

Training crawlers made 71% of requests; live fetches for a user made 16%

The 71% training share is in the same neighbourhood as what the big networks report. Cloudflare put training at "nearly 80%" of AI-bot activity in July 2025 in its crawl-to-click analysis, and its July 2026 bot report says 52% of all crawler requests were for AI training as of June 2026, up from 22% in spring 2025. A 12-site log study by Digital Applied from April 2026 found training at roughly 89%.

It is not universal, though. Alli AI's data on 69 customer sites, covering 24.4 million requests in early 2026, found the opposite: ChatGPT-User made 133,361 requests against GPTBot's 8,864 — retrieval 15× training. On our site the OpenAI-only picture is actually the same shape (ChatGPT-User out-requested GPTBot 39:1). What tips the whole dataset toward training is a single company. Take Meta out and the remaining 35,787 requests split 46% live fetch, 35% indexing, 18% training.

That's the first practical point: "AI bot traffic" is not one thing, and a total is nearly meaningless. The purpose column is the one to read.

The top 20 AI crawlers by request volume

Below the top 20 sits a long tail of 10 tokens with 90 requests between them, including DeepSeekBot (9), GrokBot (8) and MistralAI-User (8). Diffbot is worth a footnote: all 204 of its requests came on two days, August 17–18, and it never returned — a customer-triggered extraction job, not a crawler in any ongoing sense.

By company the picture is Meta 64,564, OpenAI 16,733, Microsoft 4,671, ByteDance 3,553, Google 2,941, Perplexity 2,397, Anthropic 2,265 and Amazon 1,878. Five of those eight companies run at least one live-fetch bot. Only four of them actually used one on our site: OpenAI, Anthropic, Google and DuckDuckGo.

Meta's meta-externalagent fetched every page ~120 times — 51,000 of them in eight days

Meta's crawler deserves its own section because it's the reason the totals look the way they do, and because its behaviour is unlike every other bot in the dataset.

For the first 25 days it ran at a baseline of roughly 500 requests a day. Then, from September 10 to September 17, it made 50,982 requests — about 6,400 a day, peaking at 8,990 on September 15 — from 196 IP addresses in three of Meta's network blocks. On September 16 alone it fetched the homepage 329 times, /pricing 327 times and /auth/login 312 times. On September 15 the 322 fetches of /pricing were a median of three seconds apart.

That last detail is the tell. A crawler discovering a site fetches each URL once and moves on; ClaudeBot, CCBot and YouBot all fetched roughly 80% of their URLs exactly once in this window. meta-externalagent fetched only 6 of its 537 URLs once. Averaged across the whole 35 days it hit every URL it knew about 120 times, and 10.9% of its requests — 7,051 of them — went to the login page, the privacy policy, the terms of service and the pricing page, none of which contain anything a foundation model could learn from twice.

What the crawler is for, in Meta's own words: it "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly", per Meta's web crawler documentation. Meta says it respects robots.txt, with changes taking up to 24 hours to apply. Crucially, it is not the crawler Meta ties to citations. That's meta-webindexer, which Meta describes as improving "Meta AI search result quality" — and which, as noted above, our registry didn't yet include. In 35 days our analytics recorded no referral visits from Meta AI.

Is 64% normal? It depends entirely on whose network you ask. Cloudflare's purpose-and-industry breakdown had Meta-ExternalAgent at 13.9% of crawling in the Computer & Electronics vertical in August 2025, and third-party pulls of Cloudflare Radar's API put it at 10–14% of AI-bot traffic through May–August 2026 — but that denominator includes Googlebot and Bingbot. Fastly, which excludes general search crawlers, reported Meta at 52% of AI-crawler traffic in Q2 2025, 60% in Q3 2025, and 28% in its January 2026 report. Individual sites report worse: a Norwegian legal directory documented meta-externalagent at 48% of all requests — 37 times Googlebot — in April 2026, along with 45 forced application restarts in a week, and a Cloudflare community thread from February 2026 describes the same repeated re-fetching of unchanged files, with Cloudflare staff recommending a user-agent-keyed rate limit because the crawler's IP rotation defeats IP-based limits.

So our 64% is high but not an outlier for a site of this shape. The repeat-fetch rate is the number we haven't seen published anywhere else: roughly 120 requests per URL per month, from a crawler that, as far as any documentation says, never sends a visitor back.

What ChatGPT-User actually reads: 14,076 fetches across 102 of 104 blog posts

If meta-externalagent is the noise, ChatGPT-User is the signal. OpenAI is explicit that this agent fires for user actions — someone asked ChatGPT something and it opened your page while composing the answer — and that it "is not used to determine whether content may appear in Search". A ChatGPT-User hit isn't a citation, but it's the closest thing the server logs contain to one: your page was read, by ChatGPT, on behalf of a person, at that moment. How ChatGPT picks which sources to read is a longer story, but the fetch itself is the observable end of it.

98.8% of its 14,076 requests came from IPs inside OpenAI's published chatgpt-user.json ranges, from 1,941 distinct addresses. 71.5% of them went to /blog. It touched 102 of the 104 published posts at least once, plus 12 comparison pages, 82 tool pages and the homepage (1,107 fetches — brand questions, presumably).

Three things stand out in that table. First, it's concentrated: the top ten URLs account for 58.8% of all ChatGPT-User requests, while a quarter of the URLs it touched were fetched exactly once. Second, it's data. Every post in the top ten is a statistics or comparison piece — the format that gives a model a number to quote. Third, none of it is new. Measured at the moment of each fetch, 77.6% of ChatGPT-User's blog requests went to posts that were 90–180 days old, 18.8% to posts 30–90 days old, and 0.9% to anything published in the previous 30 days. That's partly our own publishing calendar (the statistics cluster went out in April and May), so read it as "what it fetched" rather than "what it prefers" — but the newest post in the top ten was three and a half months old.

Two more behavioural details. ChatGPT-User fetched zero images, scripts or stylesheets — HTML only, 0.0% static assets — whereas Bingbot spent 22.7% of its requests on images and Open Graph cards and was the only crawler indexing the charts inside our posts. And its hour-of-day curve is almost flat, between 475 and 753 requests per UTC hour with a faint peak at noon UTC: a global population of askers rather than one timezone's office hours.

The most useful relationship in the whole dataset, though, is the one between ChatGPT-User and the two search indexes. It out-fetched OpenAI's own indexer, OAI-SearchBot, six to one — the index visits a page occasionally to decide eligibility; the user agent visits it every time someone asks. And of the 285 URLs ChatGPT-User fetched, 240 had also been crawled by Bingbot in the same window, which covers 99.5% of its requests. Only 48 requests — 0.3% — went to a URL that neither Bingbot nor OAI-SearchBot had touched. That matches Seer Interactive's finding that 87% of ChatGPT search citations matched Bing's top organic results, and it's why Bingbot being your third-busiest AI crawler isn't a curiosity: it's the index ChatGPT's search reads from.

One trend we can report but not explain: ChatGPT-User requests fell every single week, from 3,706 in the week of August 16 to 1,875 in the week of September 13 — down 49%. Over the same five weeks PerplexityBot ran 490, 494, 445, 474, 487 — flat to within 10% — and Bingbot finished up. Late August into September is a seasonal shift, OpenAI may be answering more from cached index content, or our posts may simply be getting cited less; Similarweb's July 2026 data shows ChatGPT's share of AI-platform traffic falling from around 76% to 53% over the preceding year, which would fit. We'll know more with a second month. It's the line we'd watch weekly if we watched only one.

Claude-User is mostly Claude Code, not Claude.ai

Anthropic documents Claude-User as the agent Claude uses when "individuals ask questions to Claude", per its crawler support article. We expected 1,678 of those to look like ChatGPT-User in miniature. Instead, only 21.5% of them came from Anthropic's published IP ranges, and the rest came from 1,152 different IPs spread across 1,025 different /16 network blocks — which is to say, from everywhere.

The user-agent string explains it. 1,313 of the 1,678 requests — 78% — carry Claude-User (claude-code/2.1.xxx; +https://support.anthropic.com/), with 83 distinct Claude Code versions represented and the most common (2.1.260) appearing 155 times. These are developers running Anthropic's coding agent on their own machines and asking it to read a page. They arrived at a steady 3 to 70 a day, averaging 37, and they read the same things the 360 Claude.ai fetches from Anthropic's own infrastructure read: the programmatic SEO statistics post (346 Claude Code fetches), the AI citation patterns study (265), the Reddit and AI visibility analysis (157), the SEO agency industry statistics (139) and the AI Overviews statistics (135).

That has two consequences. Practically, an IP-verification step that checks Claude-User against Anthropic's list will reject three-quarters of legitimate traffic — the requests are real, they just don't originate at Anthropic. Strategically, it's a reader segment that didn't exist eighteen months ago: people doing research through a coding agent, pulling in data-heavy pages to reason over. On this site it's already a larger source of live fetches than Google-NotebookLM and DuckAssistBot combined.

Perplexity indexes on a schedule and never fetched a page live

Perplexity runs two bots. PerplexityBot is the index crawler, "designed to surface and link websites in search results"; Perplexity-User is the live fetcher that "might visit a web page to help provide an accurate answer" when someone asks, and which Perplexity says "generally ignores robots.txt rules", per Perplexity's crawler documentation.

In 35 days PerplexityBot made 2,390 requests — 98.3% from Perplexity's published ranges — and reached 90 of the 104 posts. Its weekly totals barely moved: 490, 494, 445, 474, 487. That is a scheduled recrawl, not demand. Perplexity-User appears in the logs seven times, and all seven requests came from IP addresses outside Perplexity's published perplexity-user.json list. Verified live fetches from Perplexity: zero.

That isn't necessarily how Perplexity behaves everywhere — Digital Applied's 12-site study saw Perplexity-User above ChatGPT-User — so treat it as one site's reading. But on a site Perplexity crawls diligently, the answer-time fetcher never came, and the referral side agrees: 16 visits from Perplexity in the window, against 78 from ChatGPT.

Are the crawlers who they say they are? IP verification of 94,040 requests

Every number above rests on trusting a header anyone can forge. So we downloaded the IP-range lists each vendor publishes — OpenAI's three JSON files, Perplexity's two, Anthropic's bots.json, Bing's bingbot.json, Google's four ipranges files, DuckDuckGo's, Apple's, Common Crawl's — plus the 433 routes announced by Meta's autonomous system AS32934, all on the day of analysis, and checked every request from a crawler with a list against it. That covered 94,040 of the 100,351 requests; ByteDance, Amazon, You.com, Baidu and Diffbot don't publish a list we could fetch.

The good news is that impersonation is small: about 1% of verifiable requests, once you set aside the Claude Code traffic, which is genuine. The interesting news is where it concentrates.

Googlebot is the most-impersonated real crawler, at a third of its requests, which anyone who has run a server for long will recognise — SEO tools and scrapers have borrowed that name for years. The *-Extended tokens are a cleaner case. Google's documentation states that Google-Extended "doesn't have a separate HTTP request user agent string" — it's a robots.txt control token, and crawling happens under Google's normal agents. Apple says the same of Applebot-Extended: it "does not crawl webpages". So every one of the 19 Google-Extended and 2 Applebot-Extended requests in our logs is fake by definition, before you look at an IP.

Then there are the tourists. Fifteen IP addresses on Google Cloud and AWS cycled through between three and twelve different bot identities each — one of them presented as Baiduspider, Bingbot, CCBot, ChatGPT-User, ClaudeBot, DeepSeekBot, GPTBot, Googlebot, OAI-SearchBot, PerplexityBot and two others across 58 requests. The most likely explanation is crawler-access checkers — tools that request a page as each bot to see which ones you block — being pointed at our site. They account for 332 requests, and they'd be a rounding error in any total, but a crawler dashboard that trusts the header would list them as eleven separate companies.

215 requests per visit: what the crawlers sent back

The question underneath all of this is whether any of it turns into a person on the site. For the same 35 days, our analytics recorded 5,366 human visitors, of whom 78 arrived from ChatGPT, 34 from Gemini, 30 from Claude and 16 from Perplexity — 158 in total, 2.9% of visitors. Divide each company's AI-bot requests by the visits it referred:

The comparison column is Cloudflare's crawl-to-refer ratio, read from Cloudflare Radar's AI Insights for exactly the same 35 days, and the shape matches: Google's crawling pays for itself many times over in visits, Microsoft's sits in the same place on both scales, and the AI-native companies fetch hundreds of pages per visitor they send. Our OpenAI and Perplexity ratios are roughly half and an eighth of the web-wide figures, which is what you'd expect for a site whose content is precisely the kind of statistics AI answers quote. Our Anthropic ratio looks far better than Radar's, but that number excludes the 1,313 Claude Code fetches, which never produce a referral, so the gap is mostly about what gets counted. Radar also shows why our OpenAI mix is unusual: web-wide, GPTBot is 79% of OpenAI's crawling and ChatGPT-User 12%; on this site ChatGPT-User is 84% and GPTBot 2%.

The Meta row needs no ratio. 64,564 requests, zero visits, and no documented path by which this particular crawler's fetches become citations.

For context on why anyone tolerates 215:1, the referral studies from the last year are consistent: Previsible's analysis of 166 GA4 properties found ChatGPT responsible for 92% of trackable LLM referrals but LLM traffic under 2% of sessions in every vertical; SE Ranking measured ChatGPT at 0.32% of referral traffic across 101,574 sites in May 2026; and Ahrefs reported AI search as 0.5% of its own visits but 12.1% of its signups. Small, and disproportionately valuable — the conversion benchmarks for AI referral traffic are the other half of this arithmetic. The 2.9% we saw is in line with those, and it's the 78 ChatGPT visitors, not the 64,564 Meta requests, that the crawler logs exist to explain.

Which AI crawlers to block — and the three kinds you must never block

The purpose split gives you the decision rule. Training crawlers cost you bandwidth and, as far as any vendor documents, buy you nothing visible. Index crawlers decide whether you're eligible to be cited at all. Live fetchers are the citation happening. Block the first group if you like; never block the second or third.

If you decide to cut the training crawlers, the robots.txt block is:

User-agent: meta-externalagent
User-agent: Bytespider
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Amazonbot
Disallow: /

Two operational notes from the data. Meta rotates across ~200 IPs during a burst, so a per-IP rate limit will never trigger — key the limit on the user-agent string instead, which is exactly what Cloudflare's staff advised in that February thread. And Bytespider needs enforcement at the edge rather than a polite request; 23.8% of its requests here went to /auth/login, which no well-behaved crawler has a reason to fetch 800 times. If you're on Cloudflare, its September 15, 2026 change now blocks Training and Agent bots by default on ad-bearing pages for new domains and Free-plan sites while leaving Search bots allowed — the same three-way split this post is built on.

Whether it's worth the effort depends on scale. For us, 71,000 training requests a month is a nuisance, not a bill. For Wikimedia, bots became 65% of the most expensive requests while being 35% of pageviews; Read the Docs cut bandwidth by 75% by blocking AI crawlers after one of them pulled 73 TB in a month.

How to run an AI bot traffic analysis on your own site

Everything in this post came from six steps, and none of them need a data team.

1. Log on the server, not in the browser. AI crawlers don't run JavaScript, so a tag-based analytics tool sees none of this. Middleware, an edge function, or plain access logs all work; you need the path, the timestamp, the source IP and the raw User-Agent.

2. Classify by purpose, not just by company. OpenAI is three bots with three jobs. A dashboard that says "OpenAI: 16,733" hides the only useful fact, which is that 14,076 of them were live fetches for a user.

3. Verify the IPs. Every major vendor publishes a range list, and the ones that don't — ByteDance above all — are the ones to be most suspicious of. Expect Googlebot to be a third fake and the *-Extended tokens to be entirely fake.

4. Look at requests per URL. A crawler at 1–2 requests per URL is discovering your site; one at 120 is looping. That single ratio separated Meta from everyone else more sharply than any volume figure.

5. Join it to your referral analytics. Requests per visit, per company, over the same window, is the number that tells you whether the crawling is doing anything for you.

6. Watch the live-fetch line weekly. ChatGPT-User halved in five weeks on this site while every training crawler kept going. The training line will always be larger; the AI Answers line is the one that moves when your visibility does.

QuickSEO's AI Traffic tracking handles the first two of those out of the box — a server-side snippet, classification against the same crawler registry used here, and a per-page, per-crawler view split into AI Answers, Indexing and Training — alongside the Google Search Console and AI visibility data it already tracks for your site. See what's reading your pages.

What one site's crawler logs actually say

Three takeaways survive the caveats. The total volume of AI-bot traffic is dominated by training crawlers, and by one of them in particular, whose requests never became a visit and never, by its own documentation, become a citation. The crawlers worth watching are the small ones — ChatGPT-User at 14% of requests and Claude-User at under 2% — because they are the only lines in the log where a person was on the other end. And the header lies rarely but predictably: verify against the published ranges, distrust anything calling itself Googlebot or *-Extended, and remember that a growing share of "Claude" is a developer's coding agent reading your statistics.

Next month we'll have a second window, and the first question will be whether the ChatGPT-User decline was seasonal. If your logs show the same thing, we'd like to hear about it.