Robots.txt Checker & Validator
Check any robots.txt for errors and test which crawlers it allows: Googlebot, Bingbot and AI bots like GPTBot, ClaudeBot and PerplexityBot. Free, no sign-up.
About the Robots.txt Checker
robots.txt is the plain-text file at the root of a website that tells crawlers what they may fetch. A single wrong line can hide a whole site from Google — or from ChatGPT, Claude and Perplexity. This checker fetches a site’s robots.txt, validates every line against Google’s rules, and shows exactly which search engine and AI crawlers can reach any path, and which rule decides it.
Features
- Fetch any site’s robots.txt, or paste one to validate
- Syntax check against Google’s robots.txt rules
- Googlebot and Bingbot access for any path
- AI crawler table: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and more
- Shows the exact rule that allows or blocks each crawler
- Checks the Sitemap declaration
How to Use
- Enter a domain, or switch to “Paste robots.txt” to check a draft.
- Optionally enter a path, such as /blog/, to test that page instead of the homepage.
- Click Check robots.txt.
- Review the AI crawler table — AI search bots should normally be allowed.
- Fix any syntax errors, then re-check.
How matching works: a crawler obeys only the group that names it (falling back to the * group), and within it the longest matching path wins — on a tie, Allow beats Disallow. That is how Google and Bing read the file, and how this tool evaluates it.
Frequently Asked Questions
How do I check if my robots.txt is blocking Google?
Enter your domain and look at the Search engines table. It shows whether Googlebot and Bingbot may crawl the path you test, and the exact Allow or Disallow line that decides it. Type a different path, such as /blog/, to test a specific page.
Should I block GPTBot and other AI crawlers?
It depends on what you want. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages for AI answers — block them and your site can’t be cited there. Training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot collect data for model training; blocking those opts you out of training without affecting AI search.
Why does a crawler ignore my rules for *?
A crawler obeys only the most specific group that names it. If your robots.txt has a group for Googlebot, Googlebot ignores the User-agent: * group entirely, so any rule you want it to follow must be repeated in its own group.
What happens if my site has no robots.txt?
Crawlers treat a missing robots.txt (HTTP 404) as permission to crawl everything. That is safe, but you lose the chance to declare your sitemap. A server error (5xx) is worse: Google pauses crawling the site until robots.txt is reachable again.
Is my pasted robots.txt uploaded?
No. In “Paste robots.txt” mode the text is parsed and checked in your browser. Only “Check a website” mode contacts our server, and only to download the site’s public robots.txt.