Perplexity runs two crawlers, they do different jobs, and only one of them follows your robots.txt. Teams that block “AI crawlers” as a single category usually get this exactly backwards.
Blocking the wrong one removes you from Perplexity’s results. Blocking the other one changes almost nothing, because it isn’t listening.
Here’s what each does, what your robots.txt decides, and what it costs when the two get treated as one thing.
The Two Crawlers
They announce themselves separately and Perplexity documents them separately.
| Crawler | What it does | Obeys robots.txt |
|---|---|---|
| PerplexityBot | Builds the index that decides which sites can surface in Perplexity results | Yes |
| Perplexity-User | Visits a page on demand when someone’s question needs it | Generally not |
The second row is the one that surprises people. Perplexity’s documentation is direct about it: because a person initiated the request, Perplexity-User generally ignores robots.txt rules.
Neither one feeds a foundation model
This matters if your reason for blocking was training. Perplexity states that PerplexityBot is not used to crawl content for AI foundation models.
A block placed to keep your writing out of a training corpus is therefore aimed at the wrong target.
What Your robots.txt Decides
It decides eligibility, and only for the indexing crawler.
Perplexity’s guidance to publishers is to allow PerplexityBot and to permit its published IP ranges. Disallow it and you aren’t ranked lower in Perplexity, you’re absent from the pool it draws from.
That’s the expensive half of the mistake. The cheap half is assuming the block also stopped on-demand fetches, which it did not.
The blanket block that costs the most
A single rule aimed at every bot with “AI” in the name is the pattern that does the damage. It removes you from a surface where you could be cited, in exchange for a protection you didn’t get.
Our wider argument on what AI search optimization involves makes the same point about the other engines: crawler access is the floor, not the strategy.
How to Check What You’re Allowing
You can settle it with three checks, and none of them takes long.
- Read your robots.txt for both names. PerplexityBot and Perplexity-User are separate tokens and a rule for one says nothing about the other.
- Check for a blanket disallow: a wildcard rule can catch both without either name appearing anywhere.
- Look in your server logs. Our walkthrough of reading server logs for AI search covers which strings to grep for.
This site allows both by name, which is the arrangement we’d suggest for any brand that wants to be quotable.
Verify before you trust the label
A user agent string is text, and anything can send it. Perplexity publishes the IP ranges for both crawlers as JSON, so a hit claiming to be Perplexity can be confirmed against the list rather than believed.
Being Allowed Is Not Being Chosen
Access gets you into the pool. It doesn’t decide what gets quoted, and Perplexity’s documentation says nothing about how sources are ranked.
What we can say is what the observable behaviour points to, and we’ve set that out in how AI crawlers pick sources. On-demand retrieval leans on freshness and on how directly a passage answers the question asked.
Then measure it rather than assume
Once access is fixed, the question becomes whether Perplexity is citing you, which is a tracking job. Our guide to tracking brand mentions in Perplexity covers running that properly.
What This Does Not Tell You
Perplexity documents what its crawlers are for. It doesn’t publish how sources are chosen, so nobody outside the company can give you a weighting.
Treat any specific claim about Perplexity’s ranking factors as inference. The crawler behaviour above is documented; the selection behaviour is observed, and the two deserve different levels of confidence.
Frequently Asked Questions
Should I allow PerplexityBot
Yes, if you want to appear in Perplexity at all. It’s the crawler that builds the index Perplexity draws results from, and disallowing it takes you out of consideration rather than lowering your position.
Does blocking Perplexity-User stop Perplexity reading my pages
Generally no. Perplexity states that Perplexity-User does not follow robots.txt rules, because the fetch happens in response to a person’s question.
Does Perplexity use my content to train a model
Perplexity states that PerplexityBot is not used to crawl content for AI foundation models. If training is your concern, the crawlers to look at belong to other companies.
How do I confirm a visit came from Perplexity
You check the requesting address against the IP lists Perplexity publishes for each crawler. User agent strings can be copied by anyone, so the address is the part worth trusting.
Is being crawled enough to get cited
No. Access makes you eligible and nothing more. What earns the citation is whether a passage answers the question better than the alternatives the engine found.
Check the Two Names This Week
Open your robots.txt and search it for both crawler names. That one check settles whether you’re eligible for Perplexity, and it’s the cheapest thing on any AI visibility list.