AI Crawler Analytics for Agencies: Track Client Websites Through One API

Your client asks which AI crawlers read their website. You need a report you can add to your own platform and explain without making promises the data cannot support. I tested that workflow with two test sites, one Team API key, and a separate access check for each client.

What does an agency need from AI crawler analytics?

AI crawler analytics for agencies needs a repeatable way to collect requests and return the right report for each client. An API lets your software request that data directly. Your client portal can use the result alongside the reports you already deliver.

That's the useful part of analytics for agencies: the data fits your work. You can collect each site's report with a scheduled API request and show it in your client portal.

The example below uses one Team read key across two sites. That means two report requests with the same credential. It doesn't mean one request downloads every client's data.

You still own the client login, access checks, and presentation. Clickport handles its supported collection paths, classification, and analytics responses. I want that division to be clear before you start building. A working integration still needs someone to maintain it.

What did the two-site test show?

The controlled test returned separate reports for both sites and rejected requests outside the permitted team. I ran the current authentication and API routes locally on 7 September 2026. The database adapters supplied synthetic records, including one distinct page path for each site.

All 11 checks passed. That means the tested route behavior matched the expected results, including the seven-day date boundaries.

These are recorded local route responses with controlled inputs. They aren't live crawler measurements. The test doesn't prove production collection, database execution, or bot identity. I also checked a separate seven-day observation from Clickport's own website, which appears below.

To repeat the access test, give your test sites different page paths. Confirm each report contains only the expected path. Then request a site outside the team and repeat the check through your client portal. A successful API call alone doesn't test client privacy.

How do you collect requests from client websites?

Use a collector where the request reaches your infrastructure: the edge, a web-server log, or application code. A browser tracking script can't record a request that never executes it. Google's documentation puts the limit plainly: “not all bots can run JavaScript.”

I would choose the collection point before writing the reporting code. Otherwise, you can build a reliable report around incomplete input.

For each site, you need an ingestion key with the agent-visits permission. Keep that separate from the Team key used to read reports. The AI agent setup guide covers the supported connectors.

Check response coverage too. The nginx connector sends eligible GET/HEAD content requests with successful HTTP 2xx responses. All 8,181 stored agent rows in my seven-day owned-site sample had successful status codes. In other words, this dataset cannot tell you how many requests failed or were blocked.

Don't promise a blocked-request report from that source. Read the edge or server's broader logs when you need those failures. The crawler tracking guide explains the underlying collection problem.

How do you query each site through one API?

You can create a Team API key in Settings > API as a team owner or admin. Choose the read permission and store the key on your backend. Select the site with site_domain on each request. The report endpoint returns summary counts, agents, pages, and a time series. You can use those fields in your existing client report.

This Node.js example reads one registered site. Save it as reports.mjs. Replace the reserved test domain and set CLICKPORT_TEAM_KEY in your server environment. Never paste the key into browser code. OWASP's secrets guidance explains why credentials need restricted access and a revocation process.

async function readReport(site_domain) {
  const key = process.env.CLICKPORT_TEAM_KEY;
  if (!key) throw new Error('Set CLICKPORT_TEAM_KEY');

  const url = new URL('https://clickport.io/api/agent-visits/report');
  url.search = new URLSearchParams({
    site_domain,
    start_date: '2026-08-31',
    end_date: '2026-09-07',
    timezone: 'UTC',
  });

  const response = await fetch(url, {
    headers: { 'x-api-key': key },
    signal: AbortSignal.timeout(10000),
  });
  if (!response.ok) throw new Error('Clickport HTTP ' + response.status);
  return response.json();
}

const report = await readReport('client-a.example');
console.log(report.pages);

The end date is exclusive. These dates cover seven complete UTC days, including 6 September. Use the same timezone and window when comparing clients.

{
  "has_data": true,
  "pages": [
    { "path": "/client-a-only", "hits": 1, "cited": 1, "visitors": 0 }
  ]
}

You can list available domains with GET /api/team/sites. If your publishing system creates sites, add sites.manage and call POST /api/team/sites. An existing team domain returns created: false, unless it is pending deletion. It doesn't issue replacement keys. The API reference documents these responses and their permissions.

What do the crawler fields mean?

Keep three questions separate: which agent made the request, what purpose its name indicates, and whether its identity passed verification. The raw-hit endpoint returns nine fields. That gives you a useful record to inspect. It doesn't reveal the answer shown to the person using an AI service.

An invalid timestamp includes one outside the accepted time window. Status must be an integer from 0 to 999. Duration defaults to 0 when invalid and is limited to 0 through 86,400,000 milliseconds. These checks mean a stored value can differ from the connector's input.

Purpose and identity answer different questions

OpenAI separates GPTBot training crawls, OAI-SearchBot search crawls, and ChatGPT-User requests. Anthropic makes a similar distinction between ClaudeBot, Claude-SearchBot, and Claude-User. Perplexity documents PerplexityBot and Perplexity-User separately.

When you label the report, preserve those differences. In Clickport's classification, indexing describes index-oriented crawling and training describes training-oriented crawling. live_retrieval describes a user-triggered fetch category. None proves what happened to the content afterward.

A familiar name isn't proof of identity either. Bing warns that user-agent strings are easy to spoof. Keep the verification verdict visible. Don't silently treat unverifiable requests as verified.

What our own site's records show

From 31 August through 6 September 2026, Clickport recorded 1,244 user-retrieval requests on its own website after excluding spoofed rows. That means 324 verified requests plus 920 unverifiable requests. A further 861 rows in this purpose category were marked spoofed.

I grouped the site's stored requests by purpose and verification before applying the report formula. The calculation produces 1,244 for the API field named citations. It isn't a count of confirmed answer citations. Both citations and cited count non-spoofed live_retrieval requests; ingested counts non-spoofed indexing or training requests. These fields don't prove answer inclusion or model training.

How do you keep each client's report private?

Your backend must check that the signed-in client can read the requested website before it calls Clickport. A Team key can read multiple sites within its team. The API's team boundary therefore doesn't replace your portal's client boundary. Check both.

I tested this distinction directly. The same key could read client A and client B. The example portal then rejected client A's request for B with HTTP 403. In practice, the extra check belongs to your application.

OWASP's object authorization guidance describes this exact class of problem. A valid login doesn't authorize every object identifier a browser submits. Resolve the client's permitted domains from trusted server data.

Don't accept a client identity merely because it appears in a request parameter. Apply the same check to report downloads and each request for more rows. If you cache results, check access before returning a cached report too.

How do you keep reports current and export the detail?

You can poll the report endpoint and cache the result for the client's requested site and period. A 60-second cache is a practical starting point for a current report. It isn't a freshness guarantee. Your chosen collector can introduce its own delay before a request reaches the API.

Poll within the site's limits

A Team key gets 300 read requests per hour and five concurrent reads per resolved site. That means one site's requests use a separate bucket from another site's requests. Team-only reads have their own bucket. Reads against an All Sites view share that view's bucket.

Include the client, site, endpoint, date bounds, timezone, and filters in your cache key. Don't share one cached response merely because two requests use /report. The HTTP caching standard explains private and no-store; neither fixes an incorrect application cache key.

When the API returns HTTP 429, respect its Retry-After header. The HTTP standard permits seconds or a date. I would cap retries at three attempts, then show a refresh error. Don't retry missing permissions as though they're rate limits.

Export the requests behind the summary

You can read request detail with GET /api/agent-visits/hits. Its default page contains up to 500 rows; the maximum is 1,000. Pass the returned next_cursor unchanged into the next request. Keep the site, dates, timezone, and filters unchanged until the cursor is null.

The cursor is opaque: your code should copy it, not interpret it. These reads return Cache-Control: no-store. The export guide covers broader downloads when you need them.

This integration uses polling. Your backend owns the schedule. Keep a saved result's timestamp visible so a failed refresh doesn't look like fresh data.

What should go into the client report?

Show the recorded crawler requests, the pages they reached, and their verification status. Include the collection method and date window. Present AI referrals and goal sessions separately. Those measurements answer different questions, and a chart should preserve that difference rather than imply a continuous customer journey.

I checked AI Search sessions on the same owned site and dates as the crawler sample. The query used final session rows and excluded rows classified as bots. It returned 22 sessions. In other words, we observed site visits with an AI Search source separately from the crawler requests.

Don't divide these numbers and call the result a conversion rate. There is no event-level join showing that those fetches produced those sessions. Google's AI features guidance also distinguishes crawling, indexing, and eligibility for supporting links.

For the client, I would add a short explanation of what changed and what needs attention. Did an important page receive requests? Did collection stop? Are unverifiable requests driving a spike? Don't turn every increase into evidence that the client's SEO improved.

The AI Search traffic guide covers the referral side. Use the revenue attribution guide when you need to connect observed human visits to business outcomes.

Frequently asked questions

Does a ChatGPT-User request prove a citation?

No. OpenAI describes ChatGPT-User as handling certain user actions. A request log doesn't contain the resulting answer or show whether it included your client's URL. Confirm a citation from the answer itself or separate provider evidence.

Is Google-Extended a crawler I can count?

Google-Extended is a robots.txt control token, without its own Google HTTP user agent. Applebot-Extended works as a usage control too. An arbitrary client can send either string, so a matching log label doesn't establish that provider's identity.

Can this work without Cloudflare?

Yes, if your hosting gives you another supported collection point, such as nginx logs or server application code. The API reads the collected data. It can't create access to logs your host doesn't expose.

Does no crawler data mean the setup failed?

Not necessarily. A site can pass its connector test before a real crawler request arrives. Check the collector, the selected site, and the date range. Treat a successful connection and observed traffic as separate states.

Can I expose the Team key to a client?

Keep the Team key on your backend. It can read multiple sites within the team. Your portal must check the client's permitted site before requesting or returning a report.

How do you test this with one client site?

Start with one site whose server or edge you can configure. Confirm the collector works, request the report, and add it to the client's existing workflow. I would judge the pilot by whether the report answers a recurring client question. A rising bot count alone doesn't establish that value.

You can start a 30-day trial and connect that first client site. For the human-traffic part of your setup, the guide to switching from Google Analytics covers the migration. Add the crawler connector alongside it, then judge the report by the questions it helps you answer.