Every website has a file that tells search engine crawlers what to look at and what to ignore. That file is robots.txt, and it lives at the root of your domain. A well-crafted robots.txt saves crawl budget, prevents duplicate content issues, and keeps sensitive areas out of search results. It works alongside your XML sitemap and canonical tags to guide crawlers effectively. A poorly written one can accidentally block your entire site from being indexed.
This guide explains how robots.txt works, walks through every directive, and shows you exactly how to write and validate one. Use our free robots.txt checker to test your file before deploying.
What Is robots.txt and How Does It Work?
Robots.txt is a plain-text file that sits at yourdomain.com/robots.txt. Search engine crawlers like Googlebot and Bingbot read this file before crawling your site. It tells them which URLs or directories they are allowed or disallowed from accessing.
Important: robots.txt is a suggestion, not a firewall. Well-behaved crawlers obey it, but malicious bots and human visitors can still access anything. Never rely on robots.txt to protect sensitive data.
- Googlebot and Bingbot respect robots.txt directives as a matter of policy.
- Blocked pages can still be indexed if other pages link to them. The crawler just cannot read the content.
- Robots.txt controls crawling, not indexing. Use meta robots or X-Robots-Tag headers to control indexing directly.
Robots.txt Syntax Explained
The syntax is simple but strict. A single wrong character can accidentally block your entire site. Here is every directive you need to know.
User-agent
Identifies which crawler the rules apply to. Use User-agent: * to target all crawlers, or specify a crawler name like User-agent: Googlebot for crawler-specific rules.
User-agent: *\nUser-agent: Googlebot-Image\nDisallow: /private/Disallow
Tells the specified crawler not to access the path that follows. An empty Disallow line (with no path) means all URLs are allowed.
User-agent: *\nDisallow: /admin/\nDisallow: /search?\nDisallow: /tmp/Note: /admin/ blocks everything inside that directory, including /admin/settings and /admin/users.
Allow
Overrides a Disallow rule for a more specific path. This is useful when you want to block a directory but allow a specific file inside it.
User-agent: *\nDisallow: /blog/\nAllow: /blog/public/Allow is evaluated before Disallow when paths overlap, so the more specific rule wins.
Sitemap
Points crawlers to your XML sitemap. This is optional but strongly recommended. It helps search engines discover pages you might not want crawled but still want indexed.
Sitemap: https://yoursite.com/sitemap.xmlCrawl-delay
Some crawlers like Bing and Yandex support crawl-delay to control how frequently they hit your server. Googlebot does not support this directive.
User-agent: Bingbot\nCrawl-delay: 10Step-by-Step: Create Your robots.txt
Here is a practical workflow for building a robots.txt that works for your site.
- Check what exists. Go to
yourdomain.com/robots.txtin a browser. If nothing is there, your site is fully open to crawlers (which is fine for most sites). Use our robots.txt checker to parse any existing file. - Block non-essential paths. Identify directories that do not need to be crawled: admin panels, staging environments, internal search result pages, and temporary files.
- Allow important resources. If you block a directory, make sure search engines can still reach CSS, JavaScript, and image files inside it. Blocked CSS can prevent Google from rendering your pages properly.
- Add your sitemap. Point to your XML sitemap with the Sitemap directive.
- Validate the file. Run it through the robots.txt checker to confirm no syntax errors accidentally block critical pages.
- Upload and test. Place the file at your domain root and monitor Google Search Console for crawl errors.
Example robots.txt for a WordPress Site
User-agent: *\nDisallow: /wp-admin/\nAllow: /wp-admin/admin-ajax.php\nDisallow: /wp-includes/\nDisallow: /?s=\nDisallow: /search?\nDisallow: /tag/\nDisallow: /author/\n\nSitemap: https://yoursite.com/sitemap.xmlThis blocks admin areas, internal search results, and tag/author archives while allowing the main content to be crawled freely.
Common robots.txt Mistakes
These errors come up frequently and can have serious consequences for your rankings.
- Blocking CSS and JavaScript. Google needs to see your styles and scripts to render pages. If you block
/wp-includes/or/static/, your pages may appear broken in search results and lose ranking potential. - Blocking the entire site accidentally. A single
Disallow: /line with no User-agent targeting will block all crawlers from your entire site. - Using robots.txt to hide sensitive data. It is not a security measure. Anyone can read it in a browser by visiting yourdomain.com/robots.txt.
- Forgetting trailing slashes.
Disallow: /adminblocks both/adminand/admin/. Be precise about what you want to block. - Not testing after changes. Always validate with the robots.txt checker before deploying. A syntax error can cost you weeks of lost rankings.
robots.txt vs. Meta Robots vs. X-Robots-Tag
These three mechanisms serve different purposes and are often confused.
- robots.txt controls crawling — whether a bot can access the page at all.
- Meta robots tag controls indexing — whether a crawled page appears in search results. Use
<meta name="robots" content="noindex">in the HTML head. - X-Robots-Tag header controls indexing via HTTP headers, useful for non-HTML files like PDFs or images.
For most sites, the combination of a clean robots.txt plus meta robots on specific pages gives you full control over how search engines interact with your content.
Test Your robots.txt Now
Paste your robots.txt into the free robots.txt checker to get an instant analysis. It will flag syntax errors, show which paths are blocked, and warn you if you are accidentally blocking important resources. Fix issues before they cost you rankings.
After any change to your robots.txt, monitor Google Search Console for crawl anomalies. A clean robots.txt file is one of the simplest and most impactful technical SEO fixes you can make.