robots.txt Generator
Block the paths you want, point crawlers at your sitemap, and choose whether to block AI training bots, without blocking AI search.
Result
Generate a robots.txt that doesn’t hide you from AI
How it works, what it checks and why it matters.
- 1Fill in the fields
Or hit “Try example” to see how it works.
- 2Generate instantly
Everything runs in your browser. Nothing is sent to a server.
- 3Copy and ship
Copy the output straight into your site.
Why it matters
Blocking GPTBot, ClaudeBot or Google-Extended keeps you out of training data. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from AI answers. This generator only blocks the first kind.
A Sitemap: line helps Google, Bing and other crawlers find the URLs you want indexed.
Content Signals let you say search=yes, ai-input=yes, ai-train=no in plain terms. It’s a preference, not a block, and about 4% of top domains already publish one.
What it checks
- Disallow rules
- One Disallow line per path you enter, under User-agent: *.
- Training-bot block
- Optional groups that block GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended and Meta-ExternalAgent.
- AI search left open
- Search and user bots like OAI-SearchBot, Claude-SearchBot and PerplexityBot get no group of their own, so they follow your general rules.
- Content Signals
- An optional Content-Signal line with your search, ai-input and ai-train choices.
- Sitemap
- A Sitemap: line with your sitemap URL.
Questions
Does robots.txt stop a page being indexed?
No, it stops crawling. A blocked URL can still appear in results if other pages link to it. Use noindex instead, and leave the page crawlable so Google can see the tag.
Will blocking GPTBot remove me from ChatGPT?
No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot and ChatGPT-User, which this generator leaves on your general rules.
Do I need a separate group for Googlebot?
No. Googlebot follows the User-agent: * group unless you give it its own. A bot with its own group ignores the * rules entirely, so repeat any rules it needs.
Is it free?
Yes. Every tool is free with no signup.
Do you store what I enter?
Browser tools never leave your device. Live tools fetch the URL you enter and don’t save the result.
How is this different from the site audit?
This tool checks one thing in depth. The site audit checks everything at once and scores SEO, AI Search and Agent Readiness.
More in Generators
Tools that pair well with this one