=== AI Crawl & Citation Report ===
Contributors: mrpritchett
Tags: ai, crawlers, robots.txt, analytics, gptbot
Requires at least: 6.4
Tested up to: 7.1
Requires PHP: 7.4
Stable tag: 1.0.0
License: GPL-2.0-or-later
License URI: https://www.gnu.org/licenses/gpl-2.0.html

See which AI bots read your site, which AI engines send people to it, and get a plain-English block or allow suggestion for each bot.

== Description ==

AI companies crawl your site to train models, to build AI search indexes, and to fetch pages live when someone asks a chatbot a question. Only some of that sends anyone back. This plugin shows you the trade on one screen.

* **AI bot crawls.** Logs crawl requests from GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Amazonbot, Amzn-SearchBot, Amzn-User, Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher, Bytespider, CCBot, MistralAI-User, MistralAI-Index, MistralAI-Training, DuckAssistBot, cohere-ai, Diffbot, YouBot, AI2Bot and Timpibot.
* **Verified or claimed.** Anyone can put "GPTBot" in a user agent. Hits are checked against the IP ranges OpenAI, Anthropic, Perplexity, Amazon, Common Crawl, Mistral and DuckDuckGo publish. Forward-confirmed reverse DNS is a fallback for Amazonbot and CCBot. Hits that can't be traced to the operator (including every bot whose operator publishes no way to check) are marked "claimed".
* **AI referrals.** Logs people who arrive from ChatGPT, Perplexity, Gemini, Copilot, Claude, Le Chat, Meta AI, DeepSeek, Grok, Duck.ai, You.com and Phind, by referrer or `utm_source` (for example `utm_source=chatgpt.com`).
* **Crawl to referral, per bot.** Two separate counts side by side, for example "ClaudeBot: 6 crawl requests; Claude: 1 referred visit." Visits are counted per engine, not credited to one bot or one crawl.
* **Top pages by crawl requests vs. top pages by referred visits.** What AI reads most next to what people actually click through to.
* **Plain-English suggestions.** Allow, consider blocking, or your call, with the reason for each bot.
* **Optional robots.txt rules.** Tick the bots to disallow, or block every suggested bot with one click. Off by default.
* **Optional RSL license.** Publish your AI terms at `/license.xml` using the Really Simple Licensing standard and point to it from robots.txt. Off by default.
* **Google-Extended and Applebot-Extended** are robots.txt switches rather than crawlers, so they never appear in the log, but you can still block them here.

The report is under **Tools → AI Crawl Report**. More at [aicrawlreport.com](https://aicrawlreport.com/?utm_source=plugin&utm_medium=readme).

= Privacy: nothing is sent off-site =

Everything is stored in one table in your WordPress database and pruned after 90 days (you can choose 7 to 730). Nothing about your site, your visitors or the report is sent anywhere.

For visitors referred by AI engines the plugin stores only the time, the engine and the page. It stores no IP address, user agent or cookie, and sets nothing in the browser. For bot hits it stores the IP and user agent so the hit can be verified.

= External services =

To verify bots, the plugin downloads these public files once a day through WP-Cron. They are plain GET requests that send nothing about your site beyond what any HTTP request carries:

* OpenAI: `https://openai.com/gptbot.json`, `https://openai.com/searchbot.json`, `https://openai.com/chatgpt-user.json` ([privacy policy](https://openai.com/policies/privacy-policy/))
* Perplexity: `https://www.perplexity.com/perplexitybot.json`, `https://www.perplexity.com/perplexity-user.json` ([privacy policy](https://www.perplexity.ai/hub/legal/privacy-policy))
* Anthropic (shared by ClaudeBot, Claude-SearchBot and Claude-User): `https://claude.com/crawling/bots.json` ([privacy policy](https://www.anthropic.com/legal/privacy))
* Common Crawl: `https://index.commoncrawl.org/ccbot.json` ([privacy policy](https://commoncrawl.org/privacy-policy))
* Mistral (separate user and index lists): `https://mistral.ai/mistralai-user-ips.json`, `https://mistral.ai/mistralai-index-ips.json` ([privacy policy](https://mistral.ai/terms/#privacy-policy))
* DuckDuckGo: `https://duckduckgo.com/duckassistbot.json` ([privacy policy](https://duckduckgo.com/privacy))
* Amazon (Amazonbot, Amzn-SearchBot and Amzn-User): `https://developer.amazon.com/amazonbot/ip-addresses/`, `https://developer.amazon.com/amazonbot/searchbot-ip-addresses/`, `https://developer.amazon.com/amazonbot/live-ip-addresses/`. Amazon publishes these lists only as web pages, so the plugin reads the IP list shown on each page and ignores the rest ([privacy notice](https://www.amazon.com/gp/help/customer/display.html?nodeId=468496))

It also runs reverse and forward DNS lookups on Amazonbot and CCBot IP addresses through your server's normal DNS resolver.

These are the only outbound requests the plugin makes.

== Installation ==

1. In WordPress, go to **Plugins → Add New**, search for "AI Crawl & Citation Report" and click **Install Now**, then **Activate**. (Or upload the zip under **Plugins → Add New → Upload Plugin**.)
2. Open **Tools → AI Crawl Report**. Crawls and referred visits are logged from the moment you activate; the report fills in as they arrive.
3. Optional: tick bots to disallow in robots.txt, publish an RSL license, or change how long data is kept. Everything is off by default.

== Frequently Asked Questions ==

= Does it send my data anywhere? =

No. The log lives in one table in your own database and is never sent off your site. The only outbound requests are the daily downloads of the bots' published IP lists described under External services, and those send nothing about your site or visitors.

= Does this block bots? =

No. It can add rules to the robots.txt WordPress generates, which well-behaved bots follow. It never blocks requests itself. Bots that ignore robots.txt (and impostors using a real bot's name) won't be stopped.

= I have a robots.txt file on disk. =

A real file replaces the one WordPress generates, so the rules won't appear. The report warns you when this is the case; copy the rules into your file or remove it.

= My site uses a page cache or CDN. =

Requests served from a full-page cache never reach WordPress, so they aren't logged. Counts will be lower than your server logs. If your site sits behind a proxy or CDN, the plugin sees the proxy's IP; add the real visitor IP with the `aiccr_client_ip` filter so verification works.

= Why are some bots always "claimed"? =

The checked official documentation for Meta, ByteDance, Cohere and some others provides no supported verification source. MistralAI-Training has no published range list in Mistral's crawler documentation. These hits stay "claimed". Published lists can also be incomplete or temporarily unavailable, so a "claimed" hit is not necessarily an impostor.

= Does a referred visit mean a particular crawl got me cited? =

No. Crawl requests and referred visits are counted separately. A referred visit is a person who clicked through from an AI engine; it's counted once for that engine and shown next to each of the engine's bots (ClaudeBot, Claude-SearchBot and Claude-User all show Claude's total), not credited to one bot or one crawl. AI answers that cite your site without anyone clicking through can't be seen from your site, so they aren't counted.

= Can I add bots or engines? =

Yes, with the `aiccr_bots` and `aiccr_engines` filters.

= Is there a paid version? =

Not yet. A paid tier with specific recommendations for your site and a weekly email digest is planned; the button at the bottom of the report links to the waitlist. This plugin stays free and complete without it. To hide the button, return an empty string from the `aiccr_waitlist_url` filter.

== Screenshots ==

1. The report: AI crawl requests, referred visits and crawl requests per referred visit for the chosen period.
2. Crawl to referral per bot, with a plain-English suggestion and an optional robots.txt rule for each.
3. Top pages by crawl requests next to top pages by referred visits, and the settings for the RSL license and data retention.

== Changelog ==

= 1.0.0 =
* First public release.
* Logs AI bot crawls and verifies them against each operator's published IP ranges (or reverse DNS), marking the rest "claimed".
* Logs visits referred by AI engines by referrer or `utm_source`.
* Report with crawl-to-referral per bot, top pages, plain-English suggestions, optional robots.txt rules and an optional RSL license.
* Waitlist link for the planned paid tier.

= 0.1.0 =
* First version (not published).

== Upgrade Notice ==

= 1.0.0 =
First public release.
