About the Free Robots.txt checker Tool
Inspect the robots.txt rules and sitemap declarations of a domain.
What is the Robots.txt checker?
The Robots.txt checker inspects the robots.txt file of a domain and shows you its rules. You enter a domain, and the tool retrieves the file that search engine crawlers check first at that address, then presents the user-agent blocks, the allow and disallow directives, and any sitemap declarations it contains.
Every crawler that visits a site begins by fetching robots.txt from the domain root. That small file decides which parts of the site can be crawled and which are off limits, and it also frequently points crawlers toward the XML sitemap. Because those decisions shape how search engines treat the whole domain, reading the file with a dedicated checker is far more reliable than opening it in a browser and scanning it by eye.
What the Robots.txt checker checks
The tool accepts a domain such as example.com and retrieves the robots.txt file published at its root. It then breaks the raw file into its meaningful parts so you can read them quickly: the rules grouped by user-agent, the allow and disallow directives under each, and the sitemap lines that point crawlers to your XML sitemap files.
Interpreting the output is a matter of reading the groups. A user-agent line names the crawler the following rules apply to, with an asterisk meaning the rules apply to all crawlers. The directives beneath it list which paths are blocked and which are explicitly allowed. The sitemap lines, when present, give crawlers the location of the site's sitemap so it can be discovered even without being linked from anywhere.
Because the tool reads the file exactly as it is served, it shows the live state of your crawl rules. If a directive has been added to protect a resource, or removed by accident during an update, the checker reflects what crawlers actually see when they arrive.
Reading the file is easier when you know the small amount of syntax involved. A comment introduced by a hash character is ignored, groups are separated by blank lines, and each directive occupies one line with a colon and space after its name. Rules accumulate within a group, so a long list of disallow directives simply means each listed path is blocked for that user agent.
When both a disallow and an allow rule could apply to the same path, the most specific rule wins. That behaviour is what makes patterns such as disallowing a whole directory while still allowing a single file inside it work at all, and it is why the ordering of rules matters less than their specificity. Watching for this interaction is one of the more reliable ways to understand why a carefully written file behaves the way it does in practice.
The file also commonly carries directives that sit outside the original specification. Crawl-delay, which asks crawlers to wait between requests, and Host, a holdover from the days of single-host servers, appear in many files even though the major crawlers largely ignore them today. Recognising them for what they are, legacy suggestions rather than rules every crawler obeys, prevents a lot of confusion when you inspect a file that was written years ago.
Worth keeping in mind: robots.txt is completely public. Anyone who can reach your domain can read it, which is why it should never be used to protect private content or to hide valuable pages from human visitors. A file that relies on obscurity instead of real access control is a common misunderstanding, and one the checker surfaces simply by showing you exactly what the world can see.
Small details in the file change what it does. Paths are case-sensitive in the way your server treats them, so /Files and /files are different addresses to a crawler. A Sitemap line must be an absolute URL, and the standard spelling is what most crawlers recognise. Getting these details right is what separates a file that guides crawlers from one that quietly confuses them.
Finally, treat the file as a living document that deserves a routine. Every crawl-related change, from launching a new section to retiring an old directory, should come with a quick review of the rules. A dedicated checker fits that routine because it shows the parsed result rather than raw text, which makes it far easier to see at a glance whether the file still matches the site structure you have today.
How to use the Robots.txt checker
Inspecting a domain's crawl rules takes only a moment. Follow these steps:
- Enter the domain whose robots.txt you want to inspect, such as example.com, with or without the www prefix.
- Leave out the protocol and any path, since the tool builds the correct request for the domain root itself.
- Submit the form to fetch the robots.txt file from the server.
- Wait while the tool retrieves and parses the file into its rules and declarations.
- Read the result, checking the user-agent blocks, the allow and disallow directives, and any sitemap lines for anything that does not match your intentions.
How to get better results
- Check the exact domain that matters, and remember that example.com and www.example.com can serve different robots.txt content if your server treats them differently.
- Run the check after deploying changes to the file, since a typo in a directive can block far more of the site than you intended.
- Look at the sitemap lines carefully, because a missing or incorrect sitemap URL means crawlers may never discover the sitemap even though the file itself looks normal. An outdated line that still names an old sitemap path is a quiet reason for new pages to go unnoticed.
- Verify that blocking rules match your real site structure, and be especially careful with blanket disallows such as /images or /css, which can quietly stop resources from being crawled. Unless there is a deliberate reason to exclude them, most sites want those assets available.
- When debugging a crawl problem, combine the robots.txt check with a crawl of the affected pages, since the file controls crawling rather than indexing, and the two failures look different in the search results.
- Keep a copy of the file before you edit it, and re-check afterwards, so you can see precisely what changed from the crawler's point of view.
Why the Robots.txt checker matters
Robots.txt is one of the first files a search engine reads about your site, and its rules directly shape how much of your content gets crawled. A file that blocks the wrong paths can starve the site of coverage, while one that never declares a sitemap can leave your pages discovered slowly or not at all. Being able to inspect the file quickly turns that abstract risk into something you can actually see.
For anyone who manages a site, the value is in the routine. The file changes whenever crawl rules change, and those changes are easy to get wrong under pressure. Spot-checking it before a launch, after an update, or when rankings slip is a small habit that prevents the large, hard-to-diagnose problems that crawl misconfigurations create.
When to use the Robots.txt checker
- Before launching a new site or section, to confirm the crawl rules allow the content you actually want indexed.
- After editing the file, to verify the directives were published correctly and match what you intended.
- When pages disappear from search results without an obvious cause, to rule out a blocking rule you did not notice.
- When you add a new XML sitemap, to confirm the file declares it so crawlers can find it.
Frequently asked questions
Where does robots.txt have to live?
A standard robots.txt file must sit at the root of the domain, at the address /robots.txt. The checker fetches exactly that location, so the result reflects the file crawlers actually retrieve.
What is the difference between Allow and Disallow?
Disallow tells crawlers which paths are off limits, while Allow explicitly permits a path that a broader disallow would otherwise block. Together they let you exclude most of a directory while still letting crawlers reach specific files inside it.
Does robots.txt stop a page from being indexed?
No. It controls crawling, not indexing. A blocked page is simply never crawled, so it cannot be evaluated, while an indexed page that later becomes blocked may remain in the index until it is dropped. If you want a page gone, noindex is the appropriate signal, and the two mechanisms are often confused because the effects look similar from the outside.
What do the sitemap lines in the file do?
A Sitemap line tells crawlers where the XML sitemap is located, which helps it be discovered even when no other page links to it. The checker shows these declarations so you can confirm the file points at the right addresses.
How often do crawlers read robots.txt?
Crawlers fetch and cache the file regularly, and they re-check it periodically. Because of that caching, a change you publish may take a little time to be reflected in actual crawl behaviour, which is why it is worth re-checking the file after an update.