Blog · · Matt Pritchett
GPTBot in your logs might not be GPTBot
Any request can call itself GPTBot or ClaudeBot. Here's how to tell the real AI crawlers from the ones borrowing their names, and what to do about the ones you can't check.
Open any busy site's access log and you'll find GPTBot, ClaudeBot, PerplexityBot and a dozen other AI crawlers. Some of those requests really come from OpenAI, Anthropic and Perplexity. Some don't.
A user agent is just a line of text the client sends with each request. Anyone can put GPTBot in it: a scraper that wants to look respectable, a script that copies a well-known name, or a tool that wants to get past a rule you wrote for someone else. Counting user agents tells you what bots claim to be, not who's actually reading your pages.
How to check a bot is who it says
The companies that run the larger AI crawlers publish a way to check. There are two kinds:
- Published IP ranges. OpenAI, Anthropic, Perplexity, Amazon, Common Crawl, Mistral and DuckDuckGo publish lists of the addresses their bots crawl from. A request that says it's GPTBot and comes from an address on OpenAI's GPTBot list is GPTBot. One that comes from anywhere else isn't proven to be.
- Reverse and forward DNS. For some bots, such as Amazonbot and CCBot, you can look up the hostname of the requesting address, check it belongs to the operator's domain, then look that hostname up again and check it points back to the same address. Both halves matter: a reverse record alone can say anything its owner likes.
Many bots have neither. The checked documentation for Meta's crawlers, ByteDance's Bytespider, Cohere and some others gives no supported way to verify a request. MistralAI-Training has no published range list either. For those, the honest answer is "this request says it's that bot", and nothing more.
Verified and claimed
That's why AI Crawl & Citation Report, our free WordPress plugin, sorts every AI bot hit into one of two groups:
| Status | What it means |
|---|---|
| Verified | The address is on the operator's published list, or passes the forward-confirmed reverse DNS check. |
| Claimed | The request names the bot, but it couldn't be traced to the operator. |
"Claimed" doesn't mean "impostor". Published lists can lag behind a new address, a list download can fail for a day, and a bot whose operator publishes nothing is always claimed, however real it is. It means the request hasn't been proven, so it shouldn't be counted as if it had.
The plugin downloads those official lists once a day and checks each hit against them on your own server. Nothing about your site or your visitors is sent anywhere; the list downloads are the only requests it makes.
Why it matters for blocking
If you're deciding whether to block an AI crawler, the numbers you look at should be the real crawler's numbers. Inflated counts from look-alikes can make a bot seem heavier than it is, and a robots.txt rule won't stop them anyway: robots.txt is a request that well-behaved bots follow, and a scraper borrowing a famous name usually isn't one of those. If you do write rules, the free robots.txt generator for AI bots says what each one changes.
So the useful question isn't just "how often does GPTBot visit?" It's:
- How many of those visits are verified?
- How many people does that company's AI engine send back to the site?
The plugin puts those side by side for each bot, for example "ClaudeBot: 6 crawl requests; Claude: 1 referred visit", with a plain-English suggestion to allow, consider blocking, or leave it as your call. Referred visits are counted per engine, so ClaudeBot, Claude-SearchBot and Claude-User all show Claude's total rather than one crawl taking the credit.
A few things it can't see
- Cached pages. Requests a full-page cache or CDN answers never reach WordPress, so they aren't logged. Your counts will be lower than your server logs.
- Proxies. Behind a proxy or CDN, the plugin sees the proxy's address. Pass the real client address through the
aiccr_client_ipfilter so verification works. - Citations without clicks. An AI answer that cites your site without anyone clicking through can't be seen from your site.
If you run WordPress, you can download the plugin and see your own verified and claimed counts from the moment you activate it. For sites that aren't on WordPress, the hosted tracker is in private beta, and you can join the waitlist.