AI Crawl Report

Blog · · Matt Pritchett

ClaudeBot: what it is and how to see it on your site

ClaudeBot is a crawler run by Anthropic. Here is how to spot it in your traffic, why its user agent alone proves little, and how to compare its crawls with visits from Claude.

ClaudeBot is a web crawler run by Anthropic, the company behind Claude. You can spot it by "ClaudeBot" in the user agent of a request. That string only tells you what the visitor claims to be, not who sent it. To know whether ClaudeBot is really reading your pages, you need to check the request against something the user agent can't fake.

This post covers what Anthropic says ClaudeBot does, how to find it, what a crawl does and doesn't tell you, and how to see it next to the visits Claude sends back.

What ClaudeBot is

Anthropic says ClaudeBot collects web content that could potentially contribute to training its generative AI models. Anthropic also runs two other bots with other jobs: Claude-SearchBot, which reads pages to improve Claude's search results, and Claude-User, which fetches a page when a person asks Claude a question.

The distinction matters because the bots mean different things for you. A training crawler reads your pages. It does not send anyone to your site. The same split exists at OpenAI, where GPTBot collects pages for training and sends no visitors, while ChatGPT finds pages to cite through OAI-SearchBot. If you want the longer version of why a name in your logs can mislead, read GPTBot in your logs might not be GPTBot.

Anthropic's crawler documentation is the place to confirm the current list of its bots and what each one does.

How to spot ClaudeBot

The simplest way is the user agent. Search your server logs, or your analytics if it records bots, for the word ClaudeBot. Anything that matches is a request that says it is ClaudeBot.

That is where the certainty ends. Anyone can send a request with any user agent. A script run by someone else can call itself ClaudeBot, and your logs will record it the same way.

The stronger check is the address the request came from. Many AI companies publish the IP ranges their bots use. A hit from inside those ranges can be traced to the company. A hit from outside them can't. Anthropic is one of them: it publishes its crawlers' addresses at claude.com/crawling/bots.json, so a ClaudeBot hit can be verified.

AI Crawl Report uses two labels for this:

  • Verified: the hit matched the IP list the company publishes (with reverse DNS as a fallback for Amazonbot and CCBot).
  • Claimed: the hit used the bot's name but couldn't be traced to its company.

Claimed does not mean fake. Some companies, including Meta, ByteDance and Cohere, publish no way to check, so their hits are always claimed. A claimed hit is unproven, and that is all it is.

Crawl requests and referred visits are different numbers

Two numbers matter for each bot, and they answer different questions.

  • Crawl requests: how many pages the bot fetched from your site.
  • Referred visits: how many people clicked through to your site from that company's AI engine.

A bot can crawl you often and send nobody. A crawl also doesn't show that you were cited. Claude can mention your page in an answer without anyone clicking, and your site can't see that. A referred visit shows a person arrived. It doesn't show which crawl led to it.

So read the two numbers side by side and don't treat either as a citation count.

See both numbers in the report

AI Crawl Report puts crawl requests and referred visits next to each other for each bot, over 7, 30 or 90 days. Pick the window that matches how often you publish. Seven days shows a recent change. Ninety days smooths out a quiet week.

The screenshots below come from the hosted tracker pilot on sendwp.com, real traffic from Nov 3 to 8, 2026. The overview shows 1,958 AI bot requests over 7 days, of which 358 were verified and 1,600 were claimed, and 1 referred visit from 1 AI engine.

The dashboard for sendwp.com over 7 days: 1,958 AI bot requests, 358 verified and 1,600 claimed, and 1 referred visit from 1 AI engine.

The bot table breaks that down per bot, with verified and claimed hits and any visits from the matching AI engine in the same row.

A table of AI bots on sendwp.com, with verified and claimed request counts for each bot and the referred visits from its AI engine.

In that example, requests far outnumber visits, which is why the two columns sit together.

What the WordPress plugin does and doesn't do

On WordPress, the free plugin runs on your own server. It writes one row only when an AI bot or AI referral shows up. Bot checks and pruning run once a day in WP-Cron, and rows older than 90 days are deleted. Its data stays on your site. List downloads are the only outside requests.

Two limits to know:

  • Requests served from a full-page cache never reach WordPress, so counts can be lower than your server logs.
  • Behind a proxy or CDN, pass the real visitor address with the aiccr_client_ip filter, or the checks will see your proxy instead of the bot.

Robots.txt: the plugin can add rules, bots can ignore them

If you want to tell ClaudeBot to stay out, the plugin can add rules to the robots.txt that WordPress generates. This is off until you tick a box. For a site not on WordPress, or to see what each AI bot rule changes first, use the free robots.txt generator for AI bots.

The plugin doesn't block bots. Robots.txt is a request. Bots that follow it will stop, and bots that ignore it, along with impostors using ClaudeBot's name, won't. A real robots.txt file on disk replaces the one WordPress generates, and the report warns you if you have one.

The plugin can also serve an RSL (Really Simple Licensing) /license.xml and point to it from robots.txt. It states your terms. It doesn't enforce them.

Questions about setup are on the FAQ.

Check your own site

You can guess whether ClaudeBot visits you, or you can look. Get the free plugin for WordPress: it is free and GPL-licensed, needs no account or key, and has no limits. If you run another stack, join the waitlist for the hosted tracker. It is in private beta, and we email once when it opens to more people.

Read next

All posts

Join the waitlist for the hosted tracker

The hosted tracker for

sent to

The hosted tracker is in a private beta. Join the waitlist and we'll email you at most twice: one pre-order offer, and once when it opens to more sites. You'll confirm by email first. More about it