Which AI Bots Are Reading Your Website? Now You Can Watch Them Work

A bot wearing ChatGPT's name read one of our pages while I wrote this paragraph. That's not a guess. It happens hundreds of times a day across a subset of the sites we measure, around the clock, every time someone somewhere asks a question our content can answer.

For two years this traffic was invisible. Not small. Invisible. Bots passed half of all web traffic last year, the AI ones read your site a hundred times for every visitor they send back, and the analytics tool you're running shows you none of it. Ours didn't either. We measured what one of those reads is worth two weeks ago. This article is about the reads themselves: which bots they are, why they're on your site, how to watch them in your own logs today, and what the data is good for once you can.

Why is ChatGPT visiting your website?

ChatGPT visits your website for one of three reasons: GPTBot is collecting content that may train future models, OAI-SearchBot is indexing you for ChatGPT's search features, or ChatGPT-User is fetching a page right now because a real person just asked a question your site can answer. Three bots, one company, three completely different meanings.

That three-role split isn't an OpenAI quirk. Every major AI operator has converged on the same pattern, in their own documentation:

The definitions come straight from the operators. OpenAI documents ChatGPT-User as firing when "users ask ChatGPT or a CustomGPT a question." Anthropic says Claude "may access websites using a Claude-User agent" when individuals ask questions. Perplexity states flatly that PerplexityBot "is not used to crawl content for AI foundation models," and that Perplexity-User fetches pages when a user asks. Google sorts its crawlers into common crawlers, special-case crawlers, and user-triggered fetchers.

One company is missing from the table on purpose. Microsoft never joined the three-bot pattern: Copilot answers from the Bing index, so Bingbot is both a search crawler and an AI feeder in one, and Microsoft has published no separate Copilot live fetcher for you to count. Keep that in mind whenever you see a "complete" AI bot list: for some engines, the read side is simply not observable, by design.

Two details in that table trip people up constantly. Google-Extended never appears in your logs, because it isn't a bot: it's a robots.txt token that tells Google not to use Googlebot's crawl for Gemini training. If you're grepping for a Google-Extended user agent, you'll find nothing, forever. And Perplexity-User is documented by Perplexity itself as generally ignoring robots.txt, on the logic that a human requested the fetch. You can't block it with a text file. You can only see it in the data.

Why does the same company reading you three ways matter? Because the three reads are three different relationships. A training crawl is a one-time donation to a future model. An index crawl is a bid to appear in AI search results. A live fetch means your page is being read to a human right now, mid-conversation. That last one is the closest thing to a citation you can count, and it's the one worth watching weekly.

Your analytics has never seen any of this

The mechanics are plain: analytics scripts run in browsers. An AI bot requests your HTML, takes it, and leaves. No browser, no JavaScript execution, no tag fired, no pageview recorded. Vercel measured this at network scale in December 2024, still their latest crawler study, and put it in one sentence: "none of the major AI crawlers currently render JavaScript." ChatGPT's crawlers fetch JavaScript files, 11.5% of their requests, but never execute them.

Here's our own receipt. Clickport's tracker is JavaScript, like every web analytics tool. In the same 15-day window where our server-side agent data held over 6,000 IP-verified ChatGPT-User fetches, the tracker's bot-detection layer recorded this many ChatGPT-User pageviews: zero.

The tracker wasn't misconfigured. It was doing its job perfectly, and its job happens inside a browser that these bots never open. In other words: if your analytics runs in the browser, your AI read count today is zero, and not because nobody's reading you.

GA4 deserves its own paragraph here, because it's blind twice. First, the bots that never run JavaScript never reach it at all, same as everyone. Second, the rare bots that do fire hits get removed against the IAB Spiders and Bots List, silently: Google's own documentation says you "cannot disable known bot traffic exclusion or see how much known bot traffic was excluded." And the AI Assistants channel GA4 added in May 2026 classifies referral clicks only, excluding Google's own AI Overviews while it's at it. I wrote up what that channel does and doesn't cover separately. Short version: it's the visits column, added years after the reads column started filling up.

What AI bots read: 15 days of first-party data

Between June 29 and July 13, 2026, bots wearing AI and crawler names hit a subset of the sites we measure more than 20,000 times. Every hit went through the same pipeline: read the user agent, check the source IP against the operator's published ranges, store a verdict. Here's what that traffic claims to be doing:

Verification sorts that pile into three verdicts: 46% of hits verified against the operators' published IP ranges, 20% provably spoofed, 34% unverifiable either way. Meaning: nearly half of everything wearing a crawler's name could prove it, and one in five was definitely lying. More on the liars in a minute.

The verified column is the interesting one. The top verified agents across these sites, in order: ChatGPT-User first by a wide margin, then Bingbot, Applebot, Googlebot, PerplexityBot, Perplexity-User, GoogleOther, and OAI-SearchBot with single-digit hits.

Read that list again. The biggest verified reader of these sites isn't a search engine. It's ChatGPT answering live questions, at 10.8x the volume of verified Googlebot. Put another way: for every time Google's crawler read these sites, ChatGPT read them nearly eleven times on behalf of a person who was asking something at that moment.

The industry data says we're not an outlier. Imperva's 2026 Bad Bot Report put automated traffic at over half of all web traffic in 2025, the first year humans became the minority. In plain English: the average request hitting the average website is no longer a person. TollBit's Q4 2025 data measured one AI bot visit for every 31 human visits, up from one per 200 at the start of that year. Meaning: the AI share of your front door grew about sixfold in twelve months. And Cloudflare clocked GPTBot's request volume growing 305% in a single year while it climbed from their #9 crawler to #3. The reads are not a curiosity. They're becoming the majority use of your content.

A hundred reads per visitor

I'll keep this section short because the full study has its own article: across two of these sites, AI assistants fetched pages roughly 100 times for every visit they sent back. Cloudflare's numbers from their side of the network point the same direction, with OpenAI at 887 crawls per referral in mid-2025 and Anthropic at 50,000, and their July 2026 summary says AI crawlers request content "anywhere from a hundred to tens of thousands of times for every visitor they send back." In other words: being read is the default. Being visited is the exception.

So why watch a channel that pays back one visit per hundred reads? Because of what the visits do. In our study window, AI Search sessions converted at 3.4x the rate of every other channel combined, and Similarweb's panel data has ChatGPT referrals converting at 7.1%, second only to paid search. In plain English: the rare AI visitor arrives closer to a decision than almost anyone else you get. The reads are the leading indicator. The visits are small, late, and better-qualified than anything else that walks in. If you wait for the visits to show up in your referral report, you're reading the story weeks after it was written.

Most of the "Googlebot" on your site isn't Google

In those same 15 days, thousands of hits across the same sites claimed to be Googlebot. Google's published verification methods confirmed 11.9% of them. The other 88.1% came from IP addresses Google doesn't operate. Scrapers wear trusted names, and no name is more trusted than Googlebot.

In other words: seven out of every eight "Googlebot" hits were something else in a costume. Mostly scrapers betting, correctly, that nobody blocks Googlebot.

Verification is possible because the real operators want to be verifiable. OpenAI, Anthropic, Perplexity, and Google all publish machine-readable lists of the IP ranges their bots operate from, in what has become a de-facto standard format. You can look at one right now: openai.com/gptbot.json is a plain JSON file of CIDR prefixes, stamped with its creation time. Google documents two methods: match the IP against their published ranges, or run a reverse DNS lookup and confirm the hostname resolves back into googlebot.com or google.com.

A user-agent string is a claim. An IP range check is a verdict. We've caught cheap VPS boxes and, my personal favorite, a machine whose reverse DNS lookup came back as a "bugbounty" host, all wearing Google crawler user agents. Any count of AI or crawler traffic that skips verification, and every grep-based count does, is counting these too. If you've ever quoted a Googlebot number from a raw log, you've quoted some of them.

How to track AI bots in your server logs

You don't need any product to start, ours included. You need your access logs and one command. Here it is running against one of our own servers this morning:

$ awk -F'"' '{print $6}' /var/log/nginx/access.log \
    | grep -oE 'GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Googlebot|Applebot|Bytespider|CCBot|meta-externalagent|Amazonbot' \
    | sort | uniq -c | sort -rn

  20293 Googlebot      ← most will fail the IP check
  12120 Applebot
   1318 Bytespider
     63 ClaudeBot
     60 ChatGPT-User
     49 meta-externalagent
     23 OAI-SearchBot
     16 PerplexityBot
     13 Amazonbot
     12 Perplexity-User
      8 GPTBot
      6 CCBot
      5 Claude-User

That's six and three quarter hours of one morning: 20,293 hits claiming to be Googlebot, and everything you now know about spoofing applies to that number. Meaning: whatever your grep tells you about Googlebot, read it as a claim, not a count. Use zgrep and access.log* to include rotated logs, and split the pattern by role if you want the three counts separately: training bots (GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent), index builders (OAI-SearchBot|Claude-SearchBot|PerplexityBot|Applebot|Amazonbot), and live fetchers (ChatGPT-User|Claude-User|Perplexity-User). The live-fetcher count is your citations. If you want the citations-to-visits math, the calculator in our citation-gap article takes both numbers and places you against our measured benchmarks.

Now the honest part, because I'd rather you hear the limits from me. The grep approach stops in four places:

The count believes everyone. Your 20,293 "Googlebot" hits include the scrapers; separating them means fetching the operators' IP JSONs and checking every hit, per hit, forever, as ranges change weekly.

The logs rotate away. Most default configs keep 14 days. Your trend line dies at the rotation boundary.

The CDN eats the evidence. If Cloudflare or any cache serves the page, the request never reaches your origin log. On heavily cached sites, your grep sees a fraction of the reads. Cloudflare's own AI Crawl Control is a legitimate way to see what the CDN sees, if you're on it.

And the number connects to nothing. A count of reads in a terminal can't tell you which pages the bots read, whether those pages get cited, whether citations become visits, or what the visits were worth. That join is the whole game, and it's exactly the part a log file can't do.

What AI reads can tell you: five situations

A read count on its own is trivia. Joined to pages, engines, and outcomes, it starts answering questions people are guessing at today. Five situations where I've watched the data earn its place:

Content teams: your most-read pages are your citation surface

In our citation-gap study, the page AI read most was also the top AI landing page. The reads predicted where the people arrived. Which makes your most-AI-read pages a leading indicator: they show which content the engines are pulling into answers weeks before any referral shows up in analytics. Here's what that view looks like on our own site right now:

Two things I'd flag in our own table. Our France privacy article is the most-cited page on the site, 386 citations in 14 days, with 4 visitors to show for it. Meaning: the page AI finds most useful is one our referral report barely mentions, and yours will have one of these too. And the per-engine split matters: that second page is read by OpenAI and Anthropic in equal measure, but Huawei and ByteDance read it too, with zero citations attached. Same page, four companies, four different reasons.

Just as useful is the other end of the table. One ecommerce team wrote publicly about discovering, through server logs, that AI crawlers read their blog constantly but never touched their product pages. That's a selection step happening before any traffic exists. You can't see it in a referral report, and it decides which part of your site gets to exist in AI answers.

Marketers: close the loop or close the topic

"Should we care about AI traffic" is a budget argument in most companies right now, argued with screenshots of other people's studies. The reads give you your own answer: engine reads, citations, visits, and conversions and revenue on one screen, for your site. Maybe your funnel shows real money, like the consumer site in our study where AI sessions converted at 20.3%. Maybe it shows six visits and zeros, like ours below. Either way the argument is over, and it was your data that ended it.

The fastest version of that argument is a chart overlay: draw one engine's reads as a line next to your visitor curve for a quarter. If the reads climb while visits stay flat, you've measured the zero-click story on your own site instead of quoting someone else's deck. If both climb, you've found a channel while your competitors are still debating whether it exists. Either line beats a hunch, and you'll have it in one click instead of a log-parsing weekend.

Publishers: know your training-to-citation split

For a publisher, the split between training reads and live-answer reads is the licensing question in miniature. Training reads build someone else's asset; live fetches are your content being served to an audience, with or without attribution. The market has noticed: 79% of top news sites now block at least one AI training bot, News Corp licensed its content to OpenAI for a reported $250 million over five years, and Cloudflare blocks AI crawlers by default for new domains since July 2025. Your robots.txt policy should come from your own split, not from the mood of the week. And remember from the taxonomy: Perplexity-User ignores robots.txt by its own documentation. Blocking is a request. Measuring is how you check who honored it.

Technical SEO: crawl health for a second audience

Everything you do for Googlebot's crawl budget now has a second audience. Are AI bots hitting 404s? Vercel found ChatGPT wasting 34.8% of its fetches on 404 pages, against Googlebot's 8.2%, so the answer is probably yes and the fix is probably cheap. What's your AI-to-Googlebot read ratio? Ours is 10.8:1 in ChatGPT's favor; if yours is inverted, your content may not be surfacing in answers at all. And when you change robots.txt, the per-agent read counts show you which operators respected it, per day, instead of leaving you to trust.

Site owners: the reads panel is a spoof detector

Flip the verification layer around and it's a security instrument. The 88.1% fake-Googlebot figure isn't an abstraction: those are scrapers and scanners on your site today, wearing names your firewall rules were told to trust. A reads view with verification verdicts shows you the costume party directly, which is the starting point for deciding what to block. The same data answers both "which AI engines read us" and "who's pretending to be an AI engine while scraping us." It's one dataset with two jobs.

One funnel: ingested, cited, visited, converted

Everything above is measurable in principle by anyone with log access and patience. What didn't exist, in any web analytics platform we know of, was the join: AI reads and citations in the same product as sessions, goals, and revenue. Your log knows about the reads. Your analytics knows about the visits. Neither knows about the other.

So we built it. Clickport now ingests server-side agent hits, verifies every one against the operators' published IP ranges at ingest, classifies it by engine and intent, and joins it to the visit-and-conversion data the tracker already has. As far as I can tell, that makes it the only web analytics platform where AI reads, citations, visits, and conversions live in one dashboard. The engine funnel looks like this, with our own real numbers in it:

Notice our last column is zeros. Six AI visits, no conversions, on our own B2B site, in 14 days. I could have picked a prettier example. I'd rather show you the honest one, because the zeros are information too: they told us that on our site, AI is where due diligence happens without us, not yet a conversion channel. That's a different content strategy than "AI drives revenue," and we'd never have known which story was ours from industry averages. Notice also Carnegie Mellon, Huawei, and ByteDance quietly out-crawling every AI engine you've heard of, with zero citations to show for it. You only find out who's at the buffet by looking.

Beyond the funnel, the same data feeds a per-page engine split (the Pages table above), a chart overlay that draws any engine's reads next to your visitor curve, and verification verdicts on every hit. Agent hits don't count against your pageview quota either, which matters when the bots outnumber your visitors.

One honest cost: this data only exists on your server, so it needs a connector there. Not a tracker change, a dumb pipe: one key, one snippet, a few minutes, as a Cloudflare Worker, a WordPress plugin, or a few lines in any server. The setup guide covers each path.

What this doesn't show you

Three limits, stated plainly, because a launch article that reads like a brochure helps nobody.

AI visits that arrive without a referrer land in Direct, in every analytics tool, ours included. Some engines and apps strip the referrer, so the funnel's visited column reads low, and some ChatGPT traffic will always hide in Direct. Treat visited counts as a floor.

If a cache serves the page without touching your server, the connector never sees the hit. The WordPress plugin misses full-page-cached requests; on heavily cached sites, the Cloudflare Worker is the honest path because it sits in front of the cache.

And there's no prompt data in here. We can't tell you what question triggered a fetch, what the answer said, or how visible your brand is inside ChatGPT, and we won't pretend to: I've written about why simulated visibility scores are a vanity metric. This is the observed half: what the engines did on your site, verified. On privacy, the verification happens at ingest, the verdict is stored, and the bot's IP is discarded, same posture as the rest of Clickport.

FAQ

Why do bots visit my website?

Because most web traffic is now automated: Imperva measured bots at over half of all traffic in 2025. The visitors include search engine crawlers, AI training crawlers, AI search indexers, live AI fetchers answering user questions, and scrapers or scanners impersonating all of the above. Each has a different purpose, which is why counting them by name alone misleads.

Can AI read my website?

Yes. AI bots fetch your HTML directly, and Vercel measured the four biggest AI crawlers at around 1.3 billion fetches in a single month on their network alone. What they don't do is execute JavaScript, so content that only renders client-side is invisible to most of them, and so are your analytics tags.

Does Google Analytics show AI traffic?

Only the click side. GA4's AI Assistants channel, added May 2026, classifies visits that arrive with an AI referrer, and it excludes Google's own AI Overviews. The bot reads never appear: AI bots don't execute the GA4 tag, and the few bots that do fire hits are removed against the IAB bot list silently, with no report of what was excluded.

Should I allow AI crawlers on my website?

Decide per role, from your own data. Blocking training bots like GPTBot doesn't remove you from AI answers, since live fetchers and search indexers are separate bots; 79% of top news sites block at least one training bot while staying citable. Measure your training-to-citation split first, then write the robots.txt that matches your interest.

Is ChatGPT a web crawler?

ChatGPT is three different bots. GPTBot is a classic crawler collecting training data, OAI-SearchBot indexes for ChatGPT search, and ChatGPT-User is not a crawler at all: it fetches individual pages on demand when a user's question needs them. In our data, ChatGPT-User out-reads the other two combined by a wide margin.

How do I check which AI bots visit my website?

Grep your server access logs for the agent names (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot and peers); the commands are in this article. To trust the counts, verify hits against each operator's published IP ranges, because in our data 88.1% of "Googlebot" hits were fakes. Or use analytics with server-side agent tracking built in.

The reads were always there

Nothing in this article is new behavior. The bots were reading your site last year too. What's new is that you can watch: which engines, which pages, verified or fake, cited or ignored, and what came back. Half the internet's traffic spent two years off the books. It doesn't have to stay there.

You can try Clickport free and have the reads flowing next to your visitors in about five minutes with the setup guide. Or start with the grep above, today, on your own logs. I answer every email, so if your Googlebot turns out to be 88% costume party too, write to me. I'm collecting these.