Did you know that you could be hindering your crawling and indexing if your Robots.txt file isn’t set up properly? With this new tool, I help take out the guesswork and let you test your current or updated robots.txt file to make sure there is nothing in the way of search engines and AI from finding, crawling, and indexing your pages.
Paste your robots.txt — live or a draft you're about to publish — and check it against Google's own crawling rules before it costs you rankings. Nothing is fetched or uploaded; everything runs right here in your browser.
noindex meta tag or header instead — and don't block the page, or Google can't see the noindex.https://www.site.com/robots.txt and https://blog.site.com/robots.txt are evaluated completely separately — each host needs its own.crawl-delay differently.No crawling, no fetching, no waiting on a live URL to update. Just the file itself, checked the way Google actually reads it.
Copy it straight from your live site, or paste a draft you’re still working on — nothing is fetched, so unpublished changes work exactly the same as a live file.
User-agent groups, wildcard paths, and rule precedence are all resolved the way Google’s own documentation describes it — down to picking the most specific matching group and breaking ties in favor of Allow.
Critical issues surface first, then warnings, then opportunities — each with the exact line number and plain-English explanation of what it means and why it matters.
Drop in a specific path and pick a crawler — Googlebot, Bingbot, GPTBot, or generic — to see the exact Allow/Disallow rule that wins, and why.
Not every robots.txt quirk deserves the same panic. The analyzer scores every rule against Google’s own documented crawling behavior and buckets it into one of four tiers — so you fix what’s actually urgent first.
Catches an accidental Disallow: / or blocked CSS/JS before it costs you a launch.
Flags things like noindex:, crawl-delay, and host — fields Google quietly ignores, so you stop relying on them.
Surfaces missing sitemaps, unblocked search/filter parameters, and undecided access for GPTBot, ClaudeBot, and other AI crawlers.
Explicit pass confirmations for every check, not just a list of problems — so you know exactly what’s solid.
Test any specific path against any crawler and see the exact winning rule — before you publish, not after.
No. You paste the content in yourself, nothing is ever requested from your server. That’s intentional: it means you can test a draft you haven’t published yet just as easily as your live file.
Search Console checks the file that’s already live and only tells you if it’s readable. This tool actually explains why a given URL is blocked or allowed, flags deprecated fields, and lets you test unpublished drafts before they go live.
If your file has rules for both Googlebot and *, Googlebot only ever follows its own exact group, it never blends the two. This tool applies that same logic when it resolves a verdict.
Not reliably. Disallow stops Google from crawling a page, but if other sites link to it, it can still get indexed with no snippet. To actually remove a page from search results, use a noindex tag instead, and make sure it isn’t also disallowed, or Google can’t see the tag.
There’s no universal right answer, it’s a business call, not a technical default. The tool simply flags when it isn’t addressed at all, so it’s a deliberate decision rather than something that slipped through by accident.
No. Everything, parsing, scoring, and the URL simulator, runs entirely in your browser. Nothing is uploaded, logged, or stored anywhere.
