robots.txt Analyzer
Paste a robots.txt file to inspect its bot groups, Allow/Disallow rules, Crawl-delay directives, and Sitemap entries. The built-in simulator lets you enter a User-agent and a URL path to see whether access is allowed or blocked under the pasted rules. All analysis runs in your browser — no URLs are fetched and no content leaves your device.
Parse pasted robots.txt text, summarize crawler groups and directives, report structural warnings, and simulate whether a user-agent may access a path.
How to analyze robots.txt
- Paste a robots.txt file into the input area and run the analyzer.
- Review user-agent groups, Allow and Disallow rules, Sitemap entries, Crawl-delay values, and parser warnings.
- Use the simulator with a user-agent and URL path to see the rule selected by the tool.
Parsing and simulation rules
The parser supports User-agent, Allow, Disallow, Sitemap, Crawl-delay, and comments. Host and Clean-param are reported as non-standard rather than interpreted.
For simulation, matching groups are combined, wildcard groups are fallback, the longest matching path wins, and Allow wins an equal-length tie. The implementation also handles * wildcards and a trailing $ anchor and normalizes comparison paths in an RFC 9309-oriented form.
Privacy and limitations
robots.txt content and simulation inputs stay in your browser; the tool never requests the site or tests a real crawler. Search engines can have product-specific behavior, and Crawl-delay is parsed for display but is not part of the access decision simulated here.
Robots.txt Analyzer FAQ
Does it download my robots.txt file?
No. Paste the text manually.
Which directives are interpreted?
User-agent, Allow, Disallow, Sitemap, and Crawl-delay are parsed; comments are ignored.
What happens when two rules have the same length?
Allow wins the tie in this simulator.
Does it support * and $?
Yes. * acts as a wildcard and a trailing $ can anchor the rule to the end.
Does this guarantee how Google or another crawler behaves?
No. It is a local simulator based on the rules implemented by this tool.