How to use the robots.txt generator
- Start from a preset: allow all, block all, block admin paths or block AI training crawlers.
- Edit each group: list the user-agents, then the paths to disallow and allow (one per line).
- Add your sitemap URLs.
- Read the validation notes and test a crawler token and path to confirm the result.
- Copy or download robots.txt and upload it to the root of your site.
Worked example
Blocking WordPress admin but allowing admin-ajax
With "Disallow: /wp-admin/" and "Allow: /wp-admin/admin-ajax.php" for all crawlers, testing Googlebot on /wp-admin/admin-ajax.php shows Allowed (the longer allow rule wins), while /wp-admin/options.php shows Blocked by "Disallow: /wp-admin/".
How it works
The file is generated from your groups and parsed back with our RFC 9309 implementation: user-agent matching is case-insensitive, the most specific matching group applies (combined if repeated), the longest matching rule wins, allow wins ties, * matches any characters and $ anchors the end. Paths are percent-encoding normalised before comparison.
Assumptions
- The AI training preset uses crawler tokens verified from each operator’s documentation on 2026-09-25.
- Crawl-delay is written if you enter it, but Google ignores it.
Frequently asked questions
Does robots.txt remove pages from Google?
No. It controls crawling, not indexing. A blocked page can still be indexed if other sites link to it. To keep a page out of results, allow crawling and add a noindex meta tag or header.
Will blocking AI crawlers affect my Google rankings?
Blocking Google-Extended or other AI training tokens does not affect Google Search. Blocking Googlebot would. The preset leaves search crawlers allowed.
Do all crawlers obey robots.txt?
No. It is a voluntary standard. Reputable crawlers follow it, but it cannot technically block access, and some operators say user-triggered fetchers may not follow it.
Where do I put the file?
At the root of each host, for example https://www.example.com/robots.txt. Subdomains need their own file.
Limitations
- Validates syntax and matching logic only; it cannot tell which paths your site should block.
- Non-standard directives are flagged rather than interpreted.