Guide
Is ChatGPT Crawling Your Website?
By YourAIRank Team · Published
Short answer
OpenAI uses separate crawlers: OAI-SearchBot finds pages for ChatGPT search, GPTBot collects data for training models, and ChatGPT-User fetches a page when a user's request needs it. To check whether ChatGPT can reach your site: confirm robots.txt allows the ones you want, look for their user agents in your server or CDN logs, and verify suspicious hits against the IP ranges OpenAI publishes. Being crawled makes citations possible; it doesn't guarantee them.
OpenAI's three user agents
| User agent | What it does | If you block it |
|---|---|---|
| OAI-SearchBot | Finds and indexes pages for ChatGPT search | Your pages can't be surfaced or cited in ChatGPT search answers |
| GPTBot | Collects content that may be used to train models | Your content is excluded from future training; search is unaffected |
| ChatGPT-User | Visits a page because a user's request in ChatGPT needs it | ChatGPT can't open your page when someone asks it to |
They're controlled separately in robots.txt. Blocking GPTBot is a training decision; blocking OAI-SearchBot is a visibility decision.
Step 1: Check robots.txt
Look for rules that name the bots directly and for a broad User-agent: * rule that disallows everything — that one catches every crawler you haven't named. The robots.txt AI checker reads your file and shows which AI crawlers are allowed or blocked.
Step 2: Look for the bots in your logs
Analytics tools like GA4 won't show crawlers — bots generally don't run your tracking script. Use the raw access logs from your host or the request logs of your CDN, and filter by user agent:
What to search for in the user-agent column
- OAI-SearchBot
- GPTBot
- ChatGPT-User
Note which pages they request and whether they get a 200 response. A crawler that only ever receives 403 or 5xx errors is visiting but can't read anything.
Step 3: Verify the hits are genuine
Anyone can put "GPTBot" in a user-agent string. OpenAI publishes the IP ranges its crawlers use; compare the IP address of a suspicious request against those lists before drawing conclusions from it.
Step 4: Check firewalls and bot protection
robots.txt isn't the only gate. Some CDN, hosting and security products offer settings that block AI crawlers or challenge unknown bots. If robots.txt allows OAI-SearchBot but your logs show only blocked requests — or none at all — check those settings.
Step 5: Make sure there's something to read
A crawler that reaches a page built entirely in JavaScript may find little text. Put your main content in the HTML the server returns. The AI readiness checker flags pages with very little text before scripts run.
Crawled but still not cited?
Access is the first of three conditions. The page also has to be found for the question and contain something worth citing — see how to get cited by ChatGPT. To see whether it's happening, a ChatGPT citation tracker records which pages ChatGPT links.
Frequently asked questions
If I block GPTBot, will ChatGPT stop showing my site?
+
No. GPTBot is about training. ChatGPT search relies on OAI-SearchBot, which has its own rule.
How often does OAI-SearchBot visit?
+
There's no published schedule. Frequency varies by site; your logs are the only reliable answer for yours.
Do ChatGPT visits from people show up in analytics?
+
Yes — human clicks from ChatGPT are normal visits. See how to track ChatGPT referrals.
Start with a free check, then track it over time
The free check asks ChatGPT, Perplexity, Claude, Google AI Mode and Google AI Overviews 3 real customer questions about your business. Pro monitors up to 10 questions on ChatGPT, Perplexity, Claude, Google AI Mode and Google AI Overviews every 7 days — Pro is launching soon.