Did you know that you could be hindering your crawling and indexing if your Robots.txt file isn’t set up properly? With this new tool, I help take out the guesswork and let you test your current or updated robots.txt file to make sure there is nothing in the way of search engines and AI from finding, crawling, and indexing your pages.
Paste your robots.txt, or fetch any site's live file with one click, then check it against Google's own crawling rules before it costs you rankings. Pasting runs entirely in your browser; fetching asks our server to grab that site's robots.txt for you.
noindex meta tag or header instead, and don't block the page, or Google can't see the noindex.https://www.site.com/robots.txt and https://blog.site.com/robots.txt are evaluated completely separately, so each host needs its own.crawl-delay differently.Fetch any site’s live robots.txt in one click, or paste a draft you haven’t shipped yet. Either way, it gets checked the way Google actually reads it.
Type a domain, hit Fetch or just press Enter, and the live robots.txt loads for you. Working on something you haven’t published? Paste it into the editor instead. Both paths run through the same engine.
User-agent groups, wildcard paths, and rule precedence all resolve the way Google’s own documentation describes it, right down to picking the most specific matching group and breaking ties in favor of Allow.
Critical issues surface first, then warnings, then opportunities. Each one gives you the exact line number and a plain-English explanation of what it means and why it matters. Where there’s a clean correction, you get the fixed line to copy in one click.
Drop in a specific path or tap a quick-test chip, pick your crawler (Googlebot, Bingbot, GPTBot, or generic), and see the exact Allow or Disallow rule that wins, and why.
Not every robots.txt quirk deserves the same panic. The analyzer scores every rule against Google’s own documented crawling behavior and sorts it into one of four tiers, so you fix what’s actually urgent first and ignore what genuinely doesn’t matter.
Catches an accidental Disallow: / or blocked CSS/JS before it costs you a launch, and hands you the corrected line to copy.
Flags things like noindex:, crawl-delay, and host. Google quietly ignores these fields, so you stop relying on them to do a job they never did.
Surfaces missing sitemaps, unblocked search and filter parameters, and undecided access for GPTBot, ClaudeBot, Google-Extended, and other AI crawlers.
Explicit pass confirmations for every check, not just a list of problems, so you know exactly what’s solid before you touch anything.
Test any path against any crawler, with one-click presets for the paths people always forget, and see the exact winning rule before you publish instead of after.
Yes, if you want it to. Type your domain and hit Enter, and it pulls your live file in one click. You can also skip that entirely and paste a draft you haven’t published yet. Both routes feed the exact same analyzer, so you can check what is live today and what you are about to ship, side by side.
Search Console shows the file that is already live and mostly tells you whether it was readable. This tool explains why a given URL is blocked or allowed, flags deprecated fields, hands you the corrected line where there is one, and lets you test unpublished drafts before they ever go live.
If your file has rules for both Googlebot and *, Googlebot follows only its own exact group and never blends the two. Within that group, the longest matching rule wins, and a tie goes to Allow. The simulator applies that same order of operations and shows you which line decided the verdict.
Not reliably. Disallow stops Google from crawling a page, but if other sites link to it, it can still get indexed with no snippet. To actually remove a page from search results, use a noindex tag instead, and make sure that page is not also disallowed, or Google can never crawl it to see the tag.
There is no universal right answer. It is a business call, not a technical default. The tool only flags when the question has not been addressed at all, so allowing or blocking becomes a deliberate decision instead of something that slipped through by accident.
No. All of the parsing, scoring, and URL simulation happens in your browser, and nothing you paste is uploaded or saved. The one exception is the optional Fetch button: that sends only the domain you typed so this site can request your publicly available robots.txt on your behalf, which is required because browsers cannot read files from another domain. Nothing from that request is stored either.