robots.txt is a small text file at your domain root that tells crawlers which URLs they may fetch. It controls crawling, not indexing — a page blocked in robots.txt can still be indexed if other sites link to it. This guide covers the rules, the mistakes, and how to test your file.
How robots.txt works
Crawlers read /robots.txt before requesting any page. The file contains user-agent groups and allow or disallow rules. Googlebot, Bingbot, and AI crawlers like GPTBot and ClaudeBot all respect it. If the file is missing or empty, everything is allowed by default.
Writing correct rules
Order matters: the most specific matching rule wins. Wildcards support * for any characters and $ for end-of-line. Never block CSS or JavaScript files — Google needs them to render your pages. Block only private areas: admin panels, account pages, internal search results, and staging environments.
robots.txt versus noindex
robots.txt blocks crawling; the robots meta tag blocks indexing. If a page is blocked in robots.txt but has no noindex, search engines can still index it from external links without ever seeing your directive. To remove a page from the index, use noindex — not robots.txt.
AI crawlers
Answer engines use their own user-agents: GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot. If you want your content cited in AI answers, do not block them. Test your file against every crawler with the live tester.
Testing your file properly
A robots.txt that looks correct in a text editor can still block the wrong things. The live tester simulates each crawler's user-agent against a specific URL and returns the matching rule — catching mistakes like a Disallow: / that accidentally blocks everything, or a wildcard that matches more paths than intended. Test your homepage, your top money page, and your sitemap reference after every edit.
Common production mistakes
Four mistakes appear in nearly every broken robots.txt we audit: blocking CSS and JavaScript directories (Google needs them to render), leaving staging Disallow rules in production files, missing the Sitemap directive entirely, and creating conflicts with noindex tags on the same pages. Each is invisible until a crawler hits it — which is exactly when it becomes expensive.
AI crawler policy decisions
Decide deliberately which AI crawlers you welcome. Sites that want citations in ChatGPT, Perplexity, and AI Overviews should explicitly allow GPTBot, ClaudeBot, CCBot, Google-Extended, and PerplexityBot. Sites protecting premium content may block them — but should understand that blocking removes the site from AI answers entirely, not just from training data.
Free tools in this hub
- Robots.txt checker — Validates syntax, rules, and crawl directives on any live site
- Robots.txt generator — Builds a correct robots.txt from simple choices
- Robots.txt live tester — Tests whether a specific URL is allowed for each crawler
- Robots vs noindex conflict checker — Finds contradictions between robots.txt and meta robots
- AI robots.txt explainer — Explains what your file means for AI crawlers
Frequently asked questions
Where does robots.txt have to live?
At the domain root: https://example.com/robots.txt. A file in a subdirectory is ignored by crawlers.
Can robots.txt stop a page from being indexed?
No. It stops crawling. If other sites link to the page, search engines can still index the URL from those links. Use a noindex tag to control indexing.
Should I block AI crawlers?
Only if you do not want your content cited in AI answers. Blocking them removes your site from ChatGPT, Perplexity, and AI Overview citations entirely.
