Robots.txt Generator — Build & Validate robots.txt Free
Pick a preset or list the paths you want crawlers to skip, and get a syntax-correct robots.txt you can download and drop into your site root. One checkbox blocks the twenty user agents that scrape content for AI training. The tool flags conflicting rules and common mistakes as you type.
How it works
- 1 Choose a starting point Allow everything, block everything, or start from the WordPress or online-store preset. Presets fill in the paths that site type normally hides; you can edit them afterwards.
- 2 List your paths and sitemap Put one path per line in Disallow — /admin/, /cart, /*?sort= all work. Add exceptions under Allow, then paste your sitemap URL so search engines can find every page.
- 3 Copy or download the file Read the warnings, then hit Copy or Download. Upload the file to your site root so it answers at https://yourdomain.com/robots.txt — a subfolder will not work.
Your data stays private
All processing happens entirely in your browser. No files, text, or data are ever sent to our servers. You can disconnect from the internet and this tool will still work.
Frequently asked questions
- Where do I put the robots.txt file?
- In the root directory of your domain, so it loads at https://yourdomain.com/robots.txt. Crawlers only read that exact path — a file at /blog/robots.txt or /public/robots.txt is ignored. Each subdomain needs its own file too: blog.example.com does not inherit the rules from example.com.
- Does Disallow stop a page from appearing in Google?
- Not reliably. Disallow tells a crawler not to fetch the page, but Google can still index the URL if other sites link to it — you get a bare listing with no description. To keep a page out of search results, let the crawler fetch it and serve a noindex meta tag or X-Robots-Tag header instead.
- Should I block AI crawlers?
- It depends on what you gain from being cited. Blocking GPTBot, ClaudeBot, and Google-Extended keeps your content out of training sets, but Google-Extended also removes you from AI Overviews, and OAI-SearchBot controls whether ChatGPT can cite you at all. Publishers who want the referral traffic often block the training bots and leave the search bots alone.
- Does Crawl-delay work?
- Bing and Yandex honour it. Google ignores it completely — if Googlebot is hitting your server too hard, change the crawl rate in Search Console or return 429 responses. The directive is harmless to include, it just does nothing for Google.
- What is the difference between Allow and Disallow?
- Disallow blocks a path prefix, Allow carves an exception out of a blocked one. Crawlers apply the longest matching rule, so Disallow: /wp-admin/ together with Allow: /wp-admin/admin-ajax.php blocks the admin area but keeps the one file themes need. Listing the same path under both is a conflict, and this tool flags it.
- Can I use wildcards?
- Yes. * matches any run of characters and $ anchors the end of a URL, so Disallow: /*?sort= kills every sorted listing page and Disallow: /*.pdf$ blocks PDFs. Google, Bing, and Yandex all support both. Some smaller crawlers do not, so keep critical rules simple.