We logged every crawler that requested eight new posts after they went live. Googlebot reached all eight within eight hours. ClaudeBot took a median of five days.
New page crawl times decide when a search engine or an answer engine could first know a page exists. A page nobody has fetched can’t be indexed, quoted or cited, so the wait is the part worth measuring.
Here’s what arrived and in what order. We also cover what an IndexNow ping changed, and why a bot counter inside the site missed the first visits by up to a week.
What We Measured
Eight posts, published between 15 and 25 September 2026, each with a known publish time. We followed every request to each address until the morning of 29 September.
- Source: the web server’s access log, which records every request it answers.
- Counted: requests where a crawler names itself anywhere in its user agent.
- Excluded: requests with only an ordinary browser string, and our own checks.
- Window: between 3.8 and 14 days per post, depending on its publish date.
The first request is the earliest time each crawler asked for each page. Every figure below is that time, in hours after publication.
The Server Log Beats a Bot Counter
A counter inside the application only sees requests that reach it, and a cached page can be served without doing so. That’s the likeliest reason for what we found next.
On four of the eight posts, the counter logged Bingbot’s first visit between 147 and 174 hours after publication. The server log had it between 0 and 6 hours. The two agreed on one post and the counter recorded nothing for three.
Timing built on the counter would have said crawlers take a week. The log says otherwise.
New Page Crawl Times by Crawler
At least nineteen named crawlers requested one of the pages. This table covers the ten that matter most for search and AI visibility.
| Crawler | Posts reached (of 8) | Within 1 hour | Within 72 hours | Median first request, hours |
|---|---|---|---|---|
| YandexBot | 8 | 6 | 8 | Under 0.1 |
| Googlebot | 8 | 0 | 8 | 3.4 |
| Bingbot | 8 | 4 | 7 | 2.6 |
| OAI-SearchBot | 5 | 0 | 5 | 8.8 |
| AhrefsBot | 8 | 0 | 7 | 19.4 |
| MJ12bot | 8 | 0 | 8 | 20.3 |
| PerplexityBot | 7 | 0 | 4 | 63.2 |
| GPTBot | 4 | 0 | 1 | 89.0 |
| ClaudeBot | 7 | 0 | 3 | 128.1 |
| Amazonbot | 7 | 0 | 2 | 198.5 |
Google arrived in hours without a ping
Googlebot’s first request landed between 2.3 and 7.1 hours after publication on all eight posts. It also beat OAI-SearchBot, PerplexityBot, GPTBot, ClaudeBot and Amazonbot on every post where they appeared.
IndexNow’s own FAQ lists Amazon, Bing, Naver, Seznam.cz, Yandex and Yep as participants, and Google isn’t among them. We didn’t test what tipped Google off, so we’re not claiming a cause.
Link-index crawlers beat the AI training crawlers
AhrefsBot reached all eight posts, with a median first request at 19.4 hours. MJ12bot, the crawler behind Majestic, took 20.3. GPTBot took 89 and ClaudeBot 128.
For a link-building site that ordering is worth knowing. The crawlers that feed link indexes saw each new page days before GPTBot or ClaudeBot did.
AI crawlers vary by days
OpenAI’s search crawler reached five of the eight posts, at a median of 8.8 hours. The three it missed were the three newest, so they may still be coming.
ChatGPT-User, which fetches on demand for a live answer, reached one page, at 24 hours. PerplexityBot reached seven posts at a median of 63 hours, and Amazonbot reached seven at 198, roughly eight days.
What an IndexNow Ping Changed
Six of the eight posts have a confirmed IndexNow submission at publication: four sent automatically and two sent by us. For the other two we found no record of one.
YandexBot arrived within eight minutes on all six that were pinged. On the two with no ping on record, it arrived after 27 and 47 hours.
Bingbot was less predictable. It arrived within seven minutes on four of the six pinged posts, but took six hours on one and 113 hours on another, despite an accepted ping. A ping is an invitation, not a schedule.
Six against two isn’t a trial. Nothing was randomised, and the posts went live in different weeks. What it does show is the size of the gap, minutes against more than a day, and that it points the way IndexNow’s design predicts.
The Requests We Excluded
Of 688 requests to the eight pages, 428 carried an ordinary browser string with no crawler name. That’s 62%. They could be readers, a cache warm-up or scrapers, and a log can’t tell them apart, so none were counted.
Named crawlers can hide inside browser strings too. Google’s mobile crawler presents as an Android Chrome string with its name at the end, and OpenAI’s search crawler shows up appended to a Mac Chrome string. A filter that reads only the start of the string misses both.
Our first pass got Google wrong
Our first pass cut every user agent at 90 characters. It reported Googlebot reaching two of the eight posts, about nine days late.
The full strings said eight of eight, inside eight hours. Two passes disagreeing is how we found the truncation, and it’s why every count here matches the crawler name anywhere in the complete string.
What the Numbers Don’t Prove
Four limits, and they matter more than the table.
- One site, eight posts: that’s a case study, not a survey.
- Crawled isn’t indexed: a request shows a crawler looked, not that it kept the page.
- Names can be forged: Cloudflare fronts the site, so we couldn’t verify crawler addresses.
- Uneven windows: newer posts had less time, so slow crawlers look faster than they are.
None of these changes the ordering. The uneven windows only flatter the slowest crawlers, which is the direction that would narrow the gap, not widen it.
How to Check Your Own Site
You need the access log and the exact time each post went live.
- Pick posts: ones with a publish time you trust.
- Pull requests: every request to each address, from the log rather than a plugin.
- Match names: search for each crawler token anywhere in the full user agent.
- Take the first: the earliest hit per crawler, in hours after publication.
Check how long your log rotates before you start. Ours keeps 15 days, which is why this covers the last eight posts and not the last hundred.
For the AI-crawler side of the same log, our notes on server logs for AI search cover the user agents worth filtering for. The same habit of reading the raw log caught the fake Google referrals we measured earlier.
What to Do About It
Send the IndexNow ping when you publish. It’s cheap, and YandexBot arrived within minutes on every post it was sent for.
Don’t plan around AI crawlers finding a page inside a day. For PerplexityBot, GPTBot, ClaudeBot and Amazonbot the median ran from 63 to 198 hours.
Judge timing from the server log. A counter inside the application put Bingbot’s first visit a week late on half our posts. If crawl budget is your real worry, crawl budget optimization explains who needs to care.
Frequently Asked Questions
How long does it take Google to crawl a new page
On this site, Googlebot’s first request arrived 2.3 to 7.1 hours after publication on all eight posts. That’s one small site, and crawling a page doesn’t mean Google will index it.
Does IndexNow work for Google
Google isn’t listed among IndexNow’s participating engines, which are Amazon, Bing, Naver, Seznam.cz, Yandex and Yep. Googlebot still reached every post within eight hours here, so we can’t say what triggered it.
How long do AI crawlers take to find a new page
Longer than search engines do. Across eight posts the median first request was 8.8 hours for OAI-SearchBot, 63 for PerplexityBot, 89 for GPTBot, 128 for ClaudeBot and 198 for Amazonbot.
Why is my bot report different from my server log
It differs because a counter inside the application only sees requests that reach it, and cached pages can be served without doing so. On four of our eight posts it put Bingbot’s first visit about a week late.
How can I tell a real crawler from a fake one
You can check the crawler’s published address ranges or run a reverse DNS lookup on the visitor. We couldn’t, because Cloudflare hides visitor addresses from our log, so every name here is a claim.
Does being crawled mean my page will be cited
No. A request shows a crawler fetched the page, not that it kept, ranked or quoted it. The fetch is the only step your server can see.
Read the First Request, Not the Total
Googlebot found every new post in hours, and ClaudeBot needed days. That timing only shows up in a log that sees every request.
Run the same check on your own last eight posts before you trust a summary of them.