How to Write a robots.txt That Actually Does What You Think
The four directives that matter, the wildcards Google supports, and the one-line mistake that quietly deindexes a whole site. With a free generator.
Four directives and a plain text file. That’s the entire standard. And still, a single stray slash in this file has knocked more sites out of Google than any other line of code I can think of — usually a staging server’s Disallow: / that rode along to production on a Friday deploy.
The Robots.txt Generator builds the file for you and flags the traps while you type. But it helps to know what you’re actually telling a crawler.
Disallow is not “hide this page”
This is the misunderstanding that costs people the most. Disallow means “don’t fetch this URL.” It does not mean “don’t list it.”
If ten sites link to your blocked /pricing page, Google can index the URL anyway — it just can’t read the contents. You end up with a naked listing in the results, title guessed from the anchor text, no description, sometimes the line “No information is available for this page.”
To keep something out of the index, do the opposite of what feels right: let the crawler fetch it, and serve a noindex meta tag or an X-Robots-Tag header. A blocked page can never be seen to have a noindex tag on it. Google also dropped support for noindex inside robots.txt itself on 1 September 2019, so anything you find in an old tutorial about that is dead.
The four directives worth knowing
User-agentopens a group. Several agent lines in a row share the rules below them.Disallowblocks a path prefix. Empty value means nothing is blocked.Allowcarves an exception out of a blocked path.Sitemaptakes an absolute URL and is read independently of every group.
Crawl-delay shows up in a lot of generated files. Bing and Yandex respect it. Google ignores it outright — if Googlebot is hammering your server, the crawl rate setting in Search Console or a 429 response is what actually helps.
Longest match wins
Crawlers don’t read your file top to bottom and stop at the first hit. Google, Bing, and Yandex apply the most specific rule, measured by path length. That’s why the standard WordPress pair works:
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpThe admin area stays shut, while the one file that themes and plugins need to call stays reachable. If the two rules tie in length, Google favours Allow.
Wildcards are supported by all three major engines: * matches any run of characters, $ pins the end of a URL. So Disallow: /*?sort= kills every faceted listing variant, and Disallow: /*.pdf$ blocks PDFs without touching /pdf-guide/.
Small things that bite
Put the file at the root. https://example.com/robots.txt and nowhere else. A file in a subfolder is invisible to crawlers, and every subdomain needs its own copy — blog.example.com inherits nothing from the apex domain.
Remember it’s public. Anyone can read yours, and plenty of people do. Writing Disallow: /internal-2026-budget/ publishes the exact path you were trying to hide, gift-wrapped.
Google caps parsing at 500 kibibytes and ignores whatever comes after. Very few sites get anywhere near that, but auto-generated files full of per-URL rules occasionally do.
The AI crawler question
Since 2023 the list of user agents has grown past twenty: GPTBot, ClaudeBot, CCBot, PerplexityBot, Bytespider, and the rest. Blocking them is one checkbox in the generator.
Whether you should is genuinely a judgment call, and I don’t think there’s a clean answer. Google-Extended keeps you out of Gemini training, but it also removes you from AI Overviews. OAI-SearchBot is the one that decides whether ChatGPT can cite you at all — block it and you’re invisible there, not just untrained-on. Publishers chasing referral traffic usually block the training bots and leave the retrieval bots alone.
Build your file in the Robots.txt Generator, read the warnings it throws, then upload it to your site root and check it loads.