Robots.txt generator for AI bots
Should you block GPTBot? It depends on what the bot does. GPTBot collects pages to train models and sends no visitors. ChatGPT finds pages to cite through a different bot, OAI-SearchBot. Pick bots by what they do and you can opt out of training and still be cited.
Free, no signup. It runs in your browser: nothing you pick or paste is sent anywhere or stored.
Choose what to allow
The default rules opt out of the 13 training crawlers and leave AI search and live fetches alone. Turn on JavaScript to change them.
Your rules
# AI bots, made with https://aicrawlreport.com/tools/robots-txt-generator User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: Meta-ExternalAgent User-agent: Bytespider User-agent: CCBot User-agent: MistralAI-Training User-agent: cohere-ai User-agent: Diffbot User-agent: AI2Bot User-agent: Timpibot Disallow: /
Test your robots.txt
Testing a pasted file needs JavaScript.
What each choice changes
Model training
These collect pages to train models, and send no visitors, so blocking them costs you no referrals. GPTBot, ClaudeBot, CCBot and Bytespider are crawlers. Google-Extended and Applebot-Extended are robots.txt tokens that no crawler carries: Googlebot and Applebot still fetch your pages, and the token says whether the company may use them for AI. Google says Google-Extended doesn't affect Search inclusion or ranking.
AI search index
These build the index an AI search product answers from. OAI-SearchBot is how ChatGPT finds pages to cite, and OpenAI says sites opted out of it are not shown in ChatGPT search answers. Block this group and you drop out of those answers.
Live fetch for a user
These read one page because someone asked an AI about it. Block them and that answer can't quote or link you. Some may not follow robots.txt; the next section says which.
Where robots.txt stops working
A robots.txt file is a request that a bot can follow or ignore. A bot that follows it stops. A scraper that uses a real bot's name doesn't, and neither does a fetcher built to ignore it.
OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked. Anthropic says its bots honor robots.txt, and discourages blocking by IP address as a way to opt out.
To tell a real visit from a fake one, read how to check whether a GPTBot hit is really from OpenAI and what ClaudeBot is and how to see it on your site. To weigh a bot's crawls against the visits its engine sends back, read PerplexityBot in your logs: crawls vs. visitors sent back and why GPTBot in your logs might not be GPTBot.
How to install it
- Add the rules to the end of your robots.txt. Keep your Googlebot, Bingbot and Sitemap lines.
- On WordPress, edit the real file if there is one. A robots.txt file on disk replaces the one WordPress generates.
- Test the finished file. Paste it into the test above.
- Give it a day. OpenAI and Perplexity say a change can take up to about 24 hours.
See who actually visits
A robots.txt file says what you want. The free WordPress plugin shows what happened: how many requests each of 27 AI bots made, checked against the IP lists their companies publish, next to the visits each AI engine sent you. The data stays on your site.
On another stack, the hosted tracker is in private beta, with a waitlist.
Questions, answered
Should I block GPTBot?
Block it if you don't want your pages used to train OpenAI's models. GPTBot sends no visitors, so blocking it costs you no referrals. ChatGPT cites pages through a different bot, OAI-SearchBot, which is a separate rule.
Does blocking GPTBot remove me from ChatGPT?
No. ChatGPT search answers come from OAI-SearchBot. OpenAI says sites that opted out of OAI-SearchBot are not shown in ChatGPT search answers, so leave that one allowed if you want to be cited.
Does Google-Extended affect my rankings?
No. Google says Google-Extended does not affect a site's inclusion in Google Search or its ranking. It is a robots.txt token that no crawler carries: Googlebot still fetches your pages, and the token only decides whether Google may use them for Gemini.
How long until a robots.txt change works?
About 24 hours. OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust, and Perplexity says up to 24 hours.