robots.txt Checker
Fetch a robots.txt and see exactly which crawlers can reach a given path, plus every sitemap it declares.
Result
Test any path against Googlebot and AI bots
How it works, what it checks and why it matters.
- 1Enter a URL
A domain or page address. No install, nothing to add to your site.
- 2We fetch it live
Our servers request the page read-only, the way a crawler would.
- 3Read and fix
Get the results in seconds, with what each one means.
Why it matters
Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from those AI answers. Blocking GPTBot or ClaudeBot only opts you out of training.
Google says a URL blocked by robots.txt can still be indexed from links, just without its content. Use noindex to keep a page out.
A crawler follows only the most specific User-agent group that names it. A bot with its own group ignores your * rules entirely.
What it checks
- File and location
- Fetches /robots.txt from the site’s origin and counts its lines.
- Sitemaps
- Every Sitemap: line the file declares.
- Search bots
- Googlebot and Bingbot, tested against your path.
- AI search and user bots
- OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot.
- AI training bots
- GPTBot, ClaudeBot and Google-Extended.
- Deciding rule
- The exact rule behind each result, using RFC 9309 matching: the longest rule wins and Allow wins a tie.
Questions
Does Google-Extended affect AI Overviews?
No. AI Overviews and AI Mode use Googlebot. Google-Extended only controls use for Gemini training and grounding outside Search.
Do AI bots obey robots.txt?
The search crawlers do. OpenAI and Perplexity say robots.txt may not apply to ChatGPT-User and Perplexity-User, because a person asked for the page. Anthropic says all three of its bots honour it.
Why is a path “allowed” with no rule?
If no rule in the matching group covers the path, it’s allowed by default. The same applies when no group matches the bot at all.
Is it free?
Yes. Every tool is free with no signup.
Do you store what I enter?
Browser tools never leave your device. Live tools fetch the URL you enter and don’t save the result.
How is this different from the site audit?
This tool checks one thing in depth. The site audit checks everything at once and scores SEO, AI Search and Agent Readiness.
More in Crawling & indexing
Tools that pair well with this one