How to Track Which AI Bots Crawl Your Site (2026)

Most teams have no idea which AI systems read their site, because analytics never shows them.

Server logs are the only place AI crawler traffic appears in full. Agents don’t run JavaScript, so they never fire an analytics tag. They arrive, take what they want, and leave no trace in the dashboard you check every morning.

To track AI bots properly you need the raw request log. We instrumented this site to capture every agent request. Here’s what 70 days showed, and how to read your own.

The Short Version

• Twenty-two distinct AI agents hit this site in 70 days, across 22,966 logged requests.
• Amazon’s crawler was the single busiest, ahead of both OpenAI and Anthropic.
• Google’s AI crawler was one of the quietest, at roughly a twelfth of Amazon’s volume.
• Crawlers that train models outnumbered those answering live questions, 3,946 requests to 2,923.
• Two fifths of agent traffic could not be attributed to a named agent at all.

What 70 Days of Logs Showed

The numbers below come from this site’s own request log between 29 May and 7 August 2026. One site, business-to-business, mid-sized. Treat it as a worked example rather than an industry benchmark.

Nine further agents appeared in smaller numbers, including perplexity-browse, gemini-train, Cohere, DuckDuckGo Assist and Diffbot.

Amazon crawls harder than OpenAI or Anthropic

This was the result we did not expect. Amazon’s crawler made more requests than any other agent, ahead of Anthropic’s training crawler and OpenAI’s browser.

Amazon barely features in most AI visibility conversations. On this site’s logs it was the heaviest reader.

Google’s AI crawler was almost absent

At 227 requests, google-ai sat near the bottom of the named list. Amazon made roughly twelve times as most requests.

Google crawls plenty through Googlebot, which is a separate agent. The point is narrower: the crawler carrying an explicitly AI label was one of the quietest here.

Training crawlers outnumbered answering crawlers

Splitting the named agents by purpose gives 3,946 requests from training crawlers against 2,923 from live-answer crawlers.

Those two do different jobs. A training crawler is deciding whether your content enters a future model. A browsing crawler is fetching a page to answer a question someone is asking right now.

We covered how the two apply different filters in how AI crawlers pick sources.

Two fifths of the traffic was unattributable

Of 22,966 requests, 13,804 carried a user agent we could name. The other 9,162 did not.

Part of it is ordinary bot noise. Some is AI traffic behind a generic or absent user agent string. Any number you read about AI crawler share, including these, is drawn from the portion that identifies itself.

What the Agents Fetched

Of all logged requests, 13,259 were page views and 9,504 were fetches of a single machine-readable manifest endpoint.

Nearly two fifths of agent traffic never touched an article. It was software asking what the site offers in a structured format.

The pages they read were not the commercial ones

The most-fetched articles were explainers and tool comparisons, not service or pricing pages.

Top of the list were pages about Ahrefs brand mentions, free social listening tools, and setting up Google Alerts. Every one is informational.

That gap between what agents read and what converts is the practical finding. Your most-crawled page and your best-converting page are rarely the same page.

Why Server Logs Are the Only Complete Source

They’re the only complete source because every other tool measures something narrower. Everything else undercounts AI crawler traffic.

Analytics misses them entirely

Google Analytics and every similar tool fire from JavaScript. Crawlers don’t execute it, so an agent can read your entire site without registering a single session.

Search Console covers Google only

Useful, and scoped to one company’s crawlers. It says nothing about Anthropic, Perplexity, Amazon or ByteDance.

CDN dashboards are good but partial

Cloudflare and Fastly both report bot traffic and classify known AI agents, which is the fastest way to start. Their classification lags new agents, and anything served from cache may not reach your origin log.

How to Read Your Own Logs

To find AI crawler traffic in your logs, filter the user agent string for known agent tokens, then group by day.

Get access to the raw log

On most hosting that means /var/log/nginx/access.log or the Apache equivalent. Managed hosts usually expose it in the control panel.

Filter for the agents that matter

Grep the user agent field for the tokens the major crawlers publish: GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Amazonbot, Applebot-Extended, Bytespider, meta-externalagent and Google-Extended.

Verify before you trust the label

A user agent string is self-reported and trivial to fake. Run a reverse DNS lookup on the requesting address and confirm it resolves to the vendor’s own domain, then forward-resolve that hostname back to the same address.

Skipping this is how teams end up reporting crawler numbers inflated by scrapers wearing someone else’s name.

Group by agent and by day

Single-day counts tell you nothing. A weekly series shows whether an agent found you recently, reads you steadily, or stopped.

What to Do With the Data

Check whether your money pages are read at all

Run your top commercial URLs against the log. If agents never fetch them, no amount of on-page work will get those pages cited.

Watch for an agent that stops

A crawler that visited weekly and then went quiet usually means a technical block, not a loss of interest. Check robots.txt, your firewall rules and your CDN bot settings before assuming anything else.

Decide what you want to allow

Training crawlers and answering crawlers can be controlled separately in robots.txt. Blocking training while allowing retrieval is a legitimate position, and so is the reverse. Make it a decision rather than an accident.

Treat crawl as necessary, not sufficient

Being fetched is not being cited. Crawl data tells you that you’re readable, and nothing about whether you’re chosen. For the second question, test the prompts themselves, which we cover in SEO vs GEO.

Frequently Asked Questions

Which AI crawler visits most

On this site over 70 days, Amazon’s crawler made the most requests at 2,773, ahead of Anthropic’s training crawler and OpenAI’s browsing agent.

Your distribution will differ. The method matters more than our numbers.

Can I see AI crawlers in Google Analytics

No. Analytics runs on JavaScript and crawlers do not execute it, so agent traffic leaves no session at all.

How do I know a crawler is genuine

To verify one, run reverse DNS on the requesting address, then forward DNS on the result, confirming both point at the vendor’s domain. User agent strings alone prove nothing.

Should I block AI crawlers

It depends on which kind and what you want. Blocking training crawlers protects content from future models but does not affect whether you get cited in live answers. Blocking retrieval crawlers removes you from those answers entirely.

Does crawl volume predict citations

No. Being read is a precondition, not a result. A page can be fetched constantly and never appear in an answer.

Instrument Your Own Site This Week

Pull last month’s access log and grep it for GPTBot, ClaudeBot, PerplexityBot and Amazonbot.

Count them by agent. Then check whether your three most important commercial pages appear anywhere in the results.

Most teams find out they’ve never been fetched at all, and that is a more useful thing to learn than any benchmark.