Clickport
Start free trial
Intuitive Dashboard Real-Time Analytics Session Tracking Engagement Tracking Goal Conversions Filtering & Segments Conversion Funnels User Flows Privacy First Bot Protection
For agencies For SaaS For e-commerce For affiliates For bloggers For publishers Google Search Console Bing Webmaster Tools WordPress Shopify Tracker install SPA tracking Custom Events API reference vs Google Analytics vs Matomo vs Cloudflare
How It Works Blog Updates Documentation UTM Builder GDPR Checker GA4 Data Loss Estimator GA Legal Status About Contact Status Security

AI crawlers

AI companies crawl the web constantly: to train models, to build search indexes, and to fetch pages live during conversations. Clickport detects these crawlers, keeps them out of your visitor numbers, and shows you exactly who came and why.

Three kinds of AI visits

Not every AI crawler hit means the same thing. Clickport classifies each detected crawler into one of three intents:

  • Live Retrieval. Your page was fetched during a real AI conversation, because an assistant needed it to answer someone's question. This is the closest thing to a human reading your content through an AI.
  • Search Indexing. Crawlers building AI search indexes. Being crawled here is a precondition for showing up in AI search results.
  • Model Training. Crawlers collecting content for model training. High volume here has no connection to traffic or citations.

Where to find it

Open Settings and switch to the Bot Center tab. The AI crawler intent panel shows the total AI crawler requests in the last 30 days, a bar splitting them across the three intents, and your top crawlers by volume. See Bot Management for everything else the Bot Center reports.

AI crawler intent · 12,840
LiveLive Retrieval · fetched during real AI conversations2,311
IndexSearch Indexing · crawled for AI search indexes4,366
TrainModel Training · crawled for AI model training6,163
Top crawlers: Applebot · GPTBot · OAI-SearchBot

Which crawlers are detected

Clickport currently recognizes 54 AI crawlers by their user agent, plus AI-company traffic arriving from known datacenter ranges. The catalog is updated as new crawlers appear.

Live retrieval

ChatGPT-User Claude-User Perplexity-User meta-externalfetcher Amzn-User MistralAI-User Gemini-Deep-Research Google-NotebookLM Google-Agent kagi-fetcher cohere-ai Cohere-Command Devin NovaAct AmazonBuyForMe

Search indexing

OAI-SearchBot Claude-SearchBot PerplexityBot Amazonbot Amzn-SearchBot meta-webindexer MistralAI-Index Google-CloudVertexBot PetalBot Bravebot YouBot PhindBot Andibot DuckAssistBot Applebot

Model training

GPTBot ClaudeBot Google-Extended GoogleOther meta-externalagent FacebookBot Applebot-Extended Bytespider Doubaobot TikTokSpider imageSpider CCBot cohere-training-data-crawler DeepSeekBot GrokBot YandexAdditional AI2Bot Ai2Bot-Dolma Diffbot ERNIEBot QwenBot ChatGLM-Spider Kimibot PanguBot

How AI crawler traffic is handled

  • Never in your stats. Detected crawler requests are rejected at ingestion and never reach your visitor numbers, sessions, or any panel. They are counted separately for the Bot Center.
  • Counted in aggregate. Clickport stores per-day, per-crawler request counts. No pages, no payloads, just who and how much.
  • Kept for 90 days. Bot statistics have their own retention window, independent of your analytics data.

AI crawlers are not AI Search traffic

These are two separate things, and Clickport keeps them strictly apart:

  • AI crawlers are bots visiting your site. They live in the Bot Center and are never counted as visitors.
  • AI Search is humans clicking through to your site from AI products. Those are real visitors, and they show up in your dashboard under the AI Search channel, next to Organic Search and Social.

A model-training crawler fetching a thousand pages produces zero visitors. One person clicking a citation in an AI answer produces one AI Search visit. Watching both tells you whether being crawled ever turns into being visited.

robots.txt and blocking

Clickport observes and reports; it cannot stop a crawler from requesting your pages, because that happens on your server before any analytics runs. If you want to opt out of AI crawling, robots.txt is the right tool. Well-behaved crawlers honor it and will disappear from the Bot Center within days. Crawlers that ignore your rules keep showing up here, which makes this panel a useful compliance check on who actually respects your robots.txt.

Related

Previous Bot Management