Robots.txt Tester

Fetching another domain's robots.txt directly from a browser tab is blocked by the same-origin policy (CORS) for the vast majority of sites, since robots.txt files aren't typically served with the CORS headers a browser requires for cross-origin JavaScript access. Routing the request through this tool's own server sidesteps that restriction — the server fetches robots.txt from the target site's root, the exact same way a real search engine crawler would, and returns the raw content for testing. This server-side fetch is restricted to public http/https URLs and blocks requests to private or internal network addresses.

This tool parses the robots.txt file into its user-agent groups and their Allow/Disallow rules, then applies the same matching algorithm real crawlers use: it finds the group that applies to your chosen user-agent (an exact name match takes priority, falling back to the wildcard * group), collects every rule in that group whose pattern matches your test path, and picks the most specific one — the rule with the longest matching pattern wins, and if an Allow and a Disallow rule are equally specific, Allow wins. This mirrors the tie-breaking behavior defined in the robots.txt specification and used by major search engines like Google.

An asterisk (*) matches any sequence of characters (including none), letting a single rule cover many URLs at once — Disallow: /*.pdf$ blocks every URL ending in .pdf, regardless of what comes before it. A dollar sign ($) at the end of a pattern anchors it to the end of the URL, meaning the pattern must match all the way to the end rather than just matching a prefix — without the trailing $, Disallow: /page would also block /page-two and /pages/anything, whereas Disallow: /page$ blocks only the exact path /page.

A robots.txt file can define separate rule groups for different named user-agents, and a crawler always follows the most specific group that names it exactly, ignoring the wildcard (*) group entirely once a name-specific group exists for it — even if that specific group is more permissive than the general rules. This is a deliberate, commonly used pattern: a site might block all crawlers by default but carve out an explicit exception for one it wants to allow (or vice versa, blocking one specific bot like GPTBot while leaving general crawling open).

The most common cause is a wildcard pattern that isn't obviously related to your path at first glance — a rule like Disallow: /*?* blocks any URL containing a query string anywhere, or Disallow: /*.php$ blocks every URL ending in .php, neither of which visually resembles a normal path prefix rule. Check the 'matched rule' shown in the test result — it displays exactly which rule the tool matched and used to make its decision, which usually clarifies an otherwise-surprising result immediately.

Not necessarily — robots.txt blocks crawling (fetching the page's content), not indexing directly. A URL disallowed in robots.txt can still occasionally appear in search results (typically as a bare URL with no title or description) if other pages link to it prominently enough, since the search engine knows the URL exists even without being able to crawl its content. To reliably prevent a page from appearing in search results at all, use a noindex meta tag or HTTP header instead — but note that a page with noindex still needs to be crawlable for the search engine to actually see and respect that noindex instruction, so blocking it in robots.txt while also relying on noindex is a contradictory setup that doesn't work as intended.

A missing robots.txt (a 404 response) is treated by virtually all crawlers as an implicit 'crawl everything' — there are no rules to restrict them, so they proceed to crawl the entire site subject only to their own general crawling policies. This tool reports clearly when a robots.txt file wasn't found, distinct from finding an empty or genuinely restrictive file, so you can tell at a glance which situation you're looking at.

Some site owners choose to block AI training crawlers like GPTBot (OpenAI), Google-Extended (Google's AI training crawler, separate from Googlebot's search crawler), or CCBot (Common Crawl, used to build datasets many AI models train on) specifically to opt their content out of being used to train large language models, while still allowing normal search engine crawlers like Googlebot and Bingbot to index their site for search visibility. This is a increasingly common, deliberate distinction — blocking AI-training-specific bots without sacrificing search presence — which is exactly why checking multiple named bots individually (rather than just checking one general 'is my site crawlable' status) matters.