Free tool

Which AI bots can access your website?

Enter your domain and see, agent by agent, whether ChatGPT, Claude, Perplexity, Gemini and 14 other AI crawlers are allowed in or blocked by your robots.txt.

See if you blocked an AI search bot by mistake, which keeps your pages out of live answers
Training and answer bots are split, so you can opt out of one and keep the other
A plain-English verdict first, the full rule detail underneath

Free. No sign-up. Results in a few seconds.

robots.txt rules for 18 AI agents, read the way a compliant crawler reads them
Whether you have an llms.txt file
noindex signals on your homepage, in meta tags and headers

One request per file, nothing else crawled. Results are cached for 15 minutes, then gone.

Allowed in is the first step. Whether an assistant names you depends on what other sites say about you. We can walk through your result and where competitors are getting cited instead.

Book a strategy call

Reading your result

Three kinds of AI agent, three different jobs

One Disallow line can switch off the job you wanted to keep. This is what each group does and what blocking it costs you.

Model training

GPTBot
ClaudeBot
Google-Extended
Applebot-Extended
Meta-ExternalAgent
CCBot

Blocking them keeps your content out of future training data. It does not stop an assistant naming you, because those mentions come mostly from what other sites say.

Your call: safe to block

AI search and answers

OAI-SearchBot
Claude-SearchBot
PerplexityBot
Googlebot
Bingbot
Amazonbot
DuckAssistBot

These decide whether your pages can be retrieved and cited in live answers. If you want to be recommended, keep them allowed.

Keep allowed

User-requested fetches

ChatGPT-User
Claude-User
Perplexity-User
Meta-ExternalFetcher
MistralAI-User

They load a page when someone asks an assistant about it. Block them and an assistant cannot read your pricing or docs when a buyer asks.

Keep allowed

Fix it

What to do with a blocked result

Unblock search and user agents

Remove the Disallow lines, or the User-agent: * block, that catch the answer bots. Keep a training block if you want one.

Check the layer robots.txt cannot show

Cloudflare and similar services can block AI bots before robots.txt is read. Look for an AI-bot or bot-fight setting in your CDN.

Confirm in your logs

Your server logs show which bots actually got through. Our guide to tracking which AI bots crawl your site covers the method.

The checker reads robots.txt the way a compliant crawler does: it finds the group that names the agent, falls back to User-agent: * when none does, and lets the longest matching rule win. Some user-triggered fetchers do not follow robots.txt at all, as OpenAI and Perplexity note in their docs. For the difference between the two files, see llms.txt vs robots.txt.

Free toolkit

Three checks, in the order that matters

Access first, then mentions, then how you stack up against competitors. Each tool runs free with no sign-up.

1. Can AI read your site?

AI crawler checker

Checks your robots.txt against 18 AI agents, so you know search and answer bots can reach your pages.

You are here

2. Does AI name you?

AI visibility checker

Builds the buyer questions for your category and scores whether ChatGPT, Perplexity, Gemini and AI Overviews mention you.

Open the visibility checker

3. How do you compare?

Turns your answer counts into share of voice, net sentiment and a margin of error against named competitors.

Open the share of voice calculator

FAQ

AI crawler questions

Straight answers about robots.txt, AI bots and what this check can see. Still unsure? Book a strategy call and we will read your result with you.

How do I check which AI bots can access my website?

Enter your domain above. The checker reads your live robots.txt and applies the RFC 9309 matching rules for 18 AI agents from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon, DuckDuckGo, Mistral and Common Crawl, then shows whether each one may fetch your homepage.

Does blocking GPTBot remove my site from ChatGPT answers?

Not on its own. OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the agent that controls whether pages can appear in ChatGPT search results. Blocking GPTBot while allowing OAI-SearchBot opts out of training but keeps you eligible for search answers.

Does Google-Extended affect AI Overviews?

Google says Google-Extended does not affect a site's inclusion in Google Search. AI Overviews are part of Search, so the agent that matters for them is Googlebot.

Why does the checker say a bot is allowed when my logs show it blocked?

robots.txt is only a request. A firewall, CDN bot setting or security plugin can block a crawler that robots.txt allows, and this checker cannot see those layers. Your server or CDN logs show what actually got through.

Do I need an llms.txt file?

It is optional. llms.txt is a community proposal published at llmstxt.org. robots.txt is the file AI crawler operators document that their crawlers follow, so fix that first.

Do you store the domains people check?

No database record is kept. A result is cached for 15 minutes so repeat checks are fast, then it expires. We send one request each for robots.txt, llms.txt and your homepage, identified as BrandMentions-AICrawlerCheck.

Allowed in. Now get named.

Crawl access makes you eligible. Editorial coverage on the sources AI assistants trust is what gets your brand recommended, and that is the work we do.