Build a valid robots.txt file with a visual rule builder — pick user-agents, add Allow/Disallow rules and crawl delays, attach your sitemap URL, and copy or download the finished file. Includes one-click presets for allowing everything, blocking everything, and blocking AI crawlers.
User-agent: *
robots.txt implements the Robots Exclusion Protocol, a decades-old, informally-standardized convention (later formalized as RFC 9309) that well-behaved crawlers check before requesting pages from your site. The file is organized into groups, each starting with one or more User-agent lines identifying which crawler the group applies to, followed by Allow and Disallow rules that permit or restrict specific paths for that crawler. It's a voluntary convention rather than an enforced security boundary — reputable crawlers from search engines and major AI companies respect it, but nothing technically stops a script or a bad actor from ignoring it entirely, so robots.txt should never be relied on to protect genuinely sensitive or private content.
It's a common misconception that Disallow in robots.txt is the tool for keeping a page out of Google's search results — it isn't, at least not reliably. Disallow only stops a crawler from requesting a URL; if that URL is discovered through a link elsewhere, Google can still list the bare URL in search results without ever reading its content, since it was never allowed to crawl it. The correct tool for excluding a specific page from search results is a noindex directive placed on the page itself (a meta tag or HTTP header), which paradoxically requires that page to remain crawlable so the noindex instruction can actually be seen and honored.
When a robots.txt file contains several groups, each crawler follows only the single most specific group that names it — a dedicated Googlebot group, if one exists, completely overrides the general * group for Googlebot specifically, rather than the two being combined or merged. Within whichever single group applies, most crawlers resolve conflicting Allow and Disallow rules by following the longest (most specific) matching path rather than simply the rule listed first or last, which is why a broad Disallow can be safely paired with a more specific Allow carving out an exception underneath it.
This tool includes ready-made presets for the handful of situations most sites eventually need: Allow all is the simplest valid file, granting universal access and typically used alongside a listed sitemap; Block all applies a blanket Disallow: / to every crawler, most often used to keep a staging, development, or password-protected preview environment out of search engines entirely; and Block AI bots keeps standard search crawling open while adding explicit Disallow rules for named AI training crawlers like GPTBot, CCBot, and Google-Extended, for site owners who want to opt content out of AI training specifically without affecting their regular search visibility.
After uploading a new robots.txt to your site's root, verify it's actually being served correctly by visiting the URL directly in a browser and confirming it returns the exact plain-text content you expect with a 200 status, not a redirect, an error page, or your site's default 404 page rendered with a 200 status (a subtle but common misconfiguration). Google Search Console's URL Inspection tool can also confirm whether Googlebot specifically sees a given URL as blocked or allowed under your current file, which is a more reliable check than assuming your rules parse the way you intended just from reading them yourself.
Robots.txt Generator controls crawler access at the site level. These related tools help with the indexing and metadata decisions that go alongside it.