Robots.txt AI Crawler Checker
See which AI and search crawlers your robots.txt allows or blocks — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot, Google-Extended, Bingbot and Applebot — and whether each one is named explicitly or just falls under the default rule.
What the checker shows
We fetch your live robots.txt and evaluate it the way major crawlers do (RFC 9309). For each crawler you get one of three answers — and we never blur them:
- Explicitly allowed — robots.txt has a group for this crawler, and it may fetch your homepage.
- Explicitly blocked — robots.txt has a group for this crawler, and it disallows your homepage.
- No specific rule found — nothing names this crawler, so it follows the
*group (or, if there isn't one, everything is allowed).
We also show whether some paths are disallowed even when the homepage is allowed, and list any sitemaps your robots.txt declares.
The crawlers that matter for AI search
- OAI-SearchBot — used to show websites in ChatGPT search results. Blocking it can keep your pages out of ChatGPT's cited sources.
- GPTBot — OpenAI's crawler for content that may be used to improve its models. It is separate from OAI-SearchBot, so you can allow one and block the other.
- ClaudeBot — Anthropic's crawler.
- PerplexityBot — used by Perplexity to find and link websites in its answers.
- Googlebot — Google Search. Blocking it removes you from Google.
- Google-Extended — not a crawler but a control token: it decides whether Google may use your content for Gemini models and grounding. It does not affect Google Search rankings or crawling.
- Bingbot and Applebot — Bing (which also powers other services, including Copilot) and Apple (Siri and Spotlight).
How robots.txt rules are read
A crawler looks for a group that names its user-agent token. If one exists, it uses only that group and ignores *. If none exists, it uses *. Within a group, the longest matching rule wins, and Allow wins a tie.
Example — blocks GPTBot, allows everyone else
- User-agent: GPTBot
- Disallow: /
- User-agent: *
- Allow: /
Here GPTBot is explicitly blocked; OAI-SearchBot, ClaudeBot and the rest have no specific rule and follow *, so they're allowed.
robots.txt is a request, not access control. Well-behaved crawlers follow it; it doesn't hide pages from people or from crawlers that ignore it.
Worked example: one robots.txt, three different answers
Here is a robots.txt as many online shops have it — a few private paths closed for everyone, one AI crawler shut out, one let in:
Example robots.txt (fictional shop)
- User-agent: *
- Disallow: /admin/
- Disallow: /checkout/
- User-agent: GPTBot
- Disallow: /
- User-agent: PerplexityBot
- Allow: /
- Sitemap: https://shop.example/sitemap.xml
The checker would report:
- GPTBot — Explicitly blocked, homepage blocked. It has its own group, and that group disallows everything.
- PerplexityBot — Explicitly allowed, homepage can crawl. Note the side effect: because PerplexityBot has its own group, it ignores the
*group completely — so /admin/ and /checkout/ are not closed for it. If you add a named group, repeat the rules you still want. - OAI-SearchBot, ClaudeBot, Googlebot, Bingbot, Applebot — No specific rule found, "Follows the * rule". Their homepage can crawl, with "Some paths disallowed: /admin/, /checkout/" shown underneath.
- Sitemaps listed: https://shop.example/sitemap.xml.
So this shop blocks OpenAI's training crawler but not OpenAI's search crawler — a common and deliberate setup. If the owner had meant to keep ChatGPT search out as well, they'd need a separate OAI-SearchBot group.
How to read your results
- Start with the summary line. "Found" means rules were read. "No robots.txt (HTTP 404)" means everything is allowed. An HTTP 5xx error is the one to fix first: many crawlers pause, or treat the whole site as blocked, while robots.txt errors.
- Then the Homepage column. "Can crawl" or "Blocked" is the bottom line for each crawler; the rule column explains why.
- "No specific rule found" is not a problem by itself. It means the crawler uses your default rules. It only matters if your default rules block pages you want cited.
- Check "Some paths disallowed". A crawler can reach your homepage but still be kept out of your service, blog or location pages — often the pages AI would cite.
- Open "Show robots.txt" to see the exact file we read, in case a CDN or plugin serves something different from what you expect.
Should you block AI crawlers?
It's a business decision. If you want AI search to recommend and cite you, the search-focused crawlers (OAI-SearchBot, PerplexityBot) need access to the pages you want cited. Some sites block training-focused crawlers while allowing search crawlers — robots.txt lets you do both.
Access is only the first step. To see whether AI actually mentions you, run the free AI visibility checker, or check other technical signals with the AI readiness checker.
Frequently asked questions
Does blocking Google-Extended remove me from Google Search?
+
No. Google-Extended only controls use of your content for Gemini models and grounding. Google Search crawling is controlled by Googlebot.
My robots.txt doesn't mention GPTBot. Is it allowed?
+
It follows your * group. If * allows your pages, GPTBot is allowed; if there's no * group either, everything is allowed. The checker shows this as "No specific rule found".
What if I have no robots.txt?
+
A missing robots.txt (404) means crawlers treat everything as allowed. A robots.txt that returns a server error is different: many crawlers pause or treat the site as blocked until it works again.
Do you store the sites I check?
+
No. We only keep a salted hash of your IP address and a timestamp for rate limiting — not the website you entered.
I changed robots.txt but the checker still shows the old rules. Why?
+
We fetch the live file each time, so the usual cause is caching: a CDN, hosting cache or SEO plugin still serving the old version. Open "Show robots.txt" to see exactly what was served.
Does allowing a crawler mean AI will cite my site?
+
No. Access is a precondition, not a guarantee. Whether your pages are cited depends on relevance and quality; the free AI visibility checker shows what AI actually does.
Start with a free check, then track it over time
The free check asks ChatGPT, Perplexity, Claude, Google AI Mode and Google AI Overviews 3 real customer questions about your business. Pro monitors up to 15 questions on ChatGPT, Perplexity, Claude, Google AI Mode and Google AI Overviews every 7 days.