Robots.txt: What It Is, How It Works, and How to Write One

A robots.txt file is a plain-text file at the root of a website that tells crawlers which URLs they may request. It lives at exactly one address: https://example.com/robots.txt. Every well-behaved crawler reads it before crawling, from Googlebot to the bots behind ChatGPT and Claude.

It is also one of the easiest files to get wrong.

A single misplaced line can stop Google crawling an entire site, and a rule written years ago can quietly keep a business out of AI search answers today. This guide covers how the file works, how search engines actually interpret it, and how to write one that does what you intend.

What robots.txt does — and what it doesn't

Robots.txt controls crawling: whether a bot may fetch a URL. It does not control indexing: whether a URL can appear in search results.

Google states plainly that a URL blocked in robots.txt can still be indexed if other pages link to it. It just appears without a description, because Google was never allowed to read the page. If you want a page kept out of search results, robots.txt is the wrong tool. Use a noindex robots meta tag or an X-Robots-Tag HTTP header instead, and make sure the page is not blocked in robots.txt, or crawlers will never see the noindex.

Robots.txt also offers no security. The file is public, anyone can read it, and obeying it is voluntary.

Bots that ignore it simply ignore it.

Never list private areas there expecting them to stay hidden; protect them with authentication.

What robots.txt is good for:

• Keeping crawlers out of areas with no search value: admin screens, internal search results, cart and checkout pages, endless filter combinations.
• Saving crawl capacity on large sites, so search engines spend their time on pages that matter.
• Declaring where your XML sitemap lives.
• Choosing which AI crawlers may use your content, and for what.

The syntax in five minutes

A robots.txt file is made of groups. Each group starts with one or more User-agent lines naming the crawlers it applies to, followed by Allow and Disallow rules:

User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /admin/help/

Sitemap: https://example.com/sitemap.xml

Line by line:

• User-agent: * applies this group to every crawler that doesn't have a group of its own.
• Disallow: /admin/ means: don't crawl any URL whose path starts with /admin/.
• Disallow: /search blocks /search, /search?q=shoes and /search-results (rules are prefixes).
• Allow: /admin/help/ is an exception inside the blocked folder.
• Sitemap: gives the full URL of your sitemap. It sits outside any group and applies to every crawler.

Google supports exactly four fields: user-agent, allow, disallow and sitemap. Everything else (crawl-delay, host, noindex, clean-param) is ignored by Google, though some other crawlers honour a few of them. Bing, for example, respects crawl-delay.

A few rules of the format that trip people up:

• Paths are case-sensitive. Disallow: /Private/ does not block /private/.
• Field names are not. user-agent, User-Agent and USER-AGENT all work.
• An empty Disallow: means "allow everything". Disallow: / means "block everything". One character apart, opposite meanings.
• Two wildcards are supported. * matches any sequence of characters, and $ anchors the end of the URL. Disallow: /*.pdf$ blocks every URL ending in .pdf.
• Paths should start with / or *. Disallow: private is ambiguous and may be ignored.

How Google decides which rule wins

This is where most real-world mistakes come from. Google's documented behaviour has two steps.

Step 1: pick the group. A crawler obeys only the single group whose user-agent matches it most specifically. If there is a group for Googlebot, Googlebot follows that group and ignores the * group entirely. Groups are not combined. (If several groups name the same crawler, those are merged into one.)

This means the following file does not keep Googlebot out of /admin/:

User-agent: *
Disallow: /admin/

User-agent: Googlebot
Disallow: /tmp/

Googlebot reads only its own group, which blocks /tmp/ and nothing else. Every rule you want a named crawler to follow has to be repeated in its group.

Step 2: pick the rule. Within the group, the rule with the longest matching path wins. If an Allow and a Disallow are equally long, the less restrictive one (Allow) wins.

User-agent: *
Disallow: /shop/
Allow: /shop/sale/

For /shop/sale/boots, both rules match. /shop/sale/ is longer than /shop/, so the page is allowed. The order of rules inside the file is irrelevant; length decides.

Where the file goes, and how errors are treated

Robots.txt must be at the root of the host it controls, served as plain text:

• https://example.com/robots.txt controls https://example.com/ only.
• https://blog.example.com/ needs its own file at https://blog.example.com/robots.txt.
• http:// and https:// versions, and www and non-www, are separate hosts too, usually handled by redirecting everything to one canonical host.

How the server responds changes everything. Per Google's documentation:

• 200 OK: the rules are applied.
• 404 or another 4xx (except 429): treated as if there were no robots.txt. Every URL may be crawled.
• 5xx server error, or a timeout: Google stops crawling the site for the first 12 hours while it retries, then falls back on the last copy it successfully fetched for up to 30 days. A robots.txt that errors is far more dangerous than one that's missing.
• Redirects are followed (at least five hops).

Google also reads only the first 500 KiB of the file, and it usually caches robots.txt for up to 24 hours, so changes aren't picked up instantly.

A common failure on modern JavaScript sites is a catch-all route that answers every URL, including /robots.txt, with the app's HTML page. Crawlers can't parse rules out of HTML, so the file is effectively missing. Open your robots.txt in a browser. If you see your site's HTML page where the rules should be, that's the problem.

Copy-paste examples

Allow everything (the right default for most small sites):

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Block one folder:

User-agent: *
Disallow: /private/

Block URL parameters that create duplicate pages:

User-agent: *
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?sessionid=

Block a file type:

User-agent: *
Disallow: /*.pdf$

A typical WordPress setup (the sitemap path shown is Yoast SEO's; WordPress's built-in sitemap is /wp-sitemap.xml):

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap_index.xml

Block everything — only for staging sites, and better done with a password:

User-agent: *
Disallow: /

Robots.txt and AI crawlers

AI companies now run several crawlers each, and each one has a different job. The important split is between bots that collect training data and bots that fetch pages for AI search answers.

Blocking one doesn't block the other.

From the companies' own documentation:

• OpenAI's settings are independent. Blocking GPTBot opts you out of training but doesn't remove you from ChatGPT search, which is controlled by OAI-SearchBot. Sites that block OAI-SearchBot aren't shown in ChatGPT search answers.
• User-triggered fetchers are a special case. OpenAI says that because ChatGPT-User acts on a user's request, "robots.txt rules may not apply". Perplexity is reported to treat its user-triggered fetcher, Perplexity-User, the same way. Anthropic says its bots honour robots.txt directives, including Crawl-delay.
• Google-Extended doesn't affect Google Search. Google states that blocking it has no effect on inclusion or ranking in Search.

Choosing a policy

Most businesses want to be found and cited in AI answers, because that is where a growing share of research starts. For them, the sensible policy is to allow the search and user bots and decide separately about training:

# AI search: allowed, so the site can be cited
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

# Model training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Remember the group rule from earlier: each named AI crawler follows only its own group, so the /admin/ rule in the * group doesn't apply to the bots named above. If they need to stay out of /admin/ too, add Disallow: /admin/ to their group as well. Robots.txt is also only the first gate for AI visibility: most AI crawlers don't run JavaScript, so a site whose content is drawn in the browser looks empty to them even when robots.txt allows everything. The AI Visibility Checker tests that alongside crawler access.

Seven mistakes that hide sites

1. A leftover Disallow: / from staging. The single most damaging robots.txt error: the whole site stops being crawled. Check the file after every launch and migration.
2. Blocking CSS and JavaScript. Google renders pages like a browser. If it can't load your stylesheets and scripts, it can't see your layout or content properly.
3. Using robots.txt to remove pages from Google. Blocked pages can still be indexed, and a blocked page's noindex tag is never seen. Unblock it and use noindex.
4. Forgetting the group rule. Adding a Googlebot group makes Googlebot ignore everything under *.
5. Case mistakes. /Blog/ and /blog/ are different paths.
6. A robots.txt that returns errors. A 5xx response pauses crawling. A missing file is safer than a broken one.
7. Blocking AI search bots by accident. Broad rules copied from a "block all AI" template often catch OAI-SearchBot and PerplexityBot too, removing the site from AI answers it might otherwise appear in.

How to test your robots.txt

Before and after every change:

1. Open https://yourdomain.com/robots.txt in a browser and confirm it shows plain-text rules.
2. Run it through the Robots.txt Checker. It validates every line against Google's rules, shows whether Googlebot, Bingbot and each AI crawler can reach any path you enter, and names the exact rule that decides it. You can also paste a draft to test it before uploading.
3. Check that the sitemap you declare actually works with the XML Sitemap Checker.
4. For the full picture (crawlability, indexing signals, speed and AI search readiness), run a Website Auditor report on your homepage.

Google Search Console also has a robots.txt report under Settings that shows the version Google last fetched and any parsing problems it found. If you pay for an SEO suite, its site audit will flag robots.txt problems as well; our Ahrefs vs Semrush vs Moz comparison covers what each one includes.

Once crawlers can reach the right pages, the next question is whether those pages deserve to rank. That's what a content audit answers: which pages to keep, update, merge or remove.

Frequently asked questions

Do I need a robots.txt file?

No. Without one, crawlers assume they may crawl everything, which is fine for many small sites. A minimal file is still worth having, if only to declare your sitemap and avoid 404s in your server logs.

Does robots.txt affect SEO?

Indirectly. It doesn't change rankings by itself, but it decides what search engines can read. Blocking important pages, stylesheets or scripts can hurt how those pages are understood and ranked.

How long do changes take to work?

Google usually caches robots.txt for up to 24 hours, so expect about a day. Other crawlers vary.

Can I block a single page?

Yes, with Disallow: /path/to/page. Remember it may still be indexed if other sites link to it. To keep it out of search results, use noindex instead.

Should I block AI crawlers?

Block training crawlers if you don't want your content used to train models. Think carefully before blocking AI search crawlers: they're how your site gets cited in ChatGPT, Claude and Perplexity answers.