Robots.txt Generator

Build a robots.txt with presets, validate it with an RFC 9309 parser and test any URL path before you publish.

  • Runs in your browser
  • Free, no sign-up
Presets

The AI training preset uses tokens verified from each operator’s documentation on 2026-09-25: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, meta-externalagent, Amazonbot, MistralAI-Training.

Group 1

Use / to block everything. * and $ are supported.

Overrides a shorter disallow rule.

Google ignores crawl-delay. Some other crawlers honour it.

robots.txt

User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /admin/
Disallow: /login
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?s=

Sitemap: https://www.example.com/sitemap.xml

Validation

No problems found by the RFC 9309 parser

Test a URL path

BlockedBlocked by "Disallow: /wp-admin/" (line 3) in the * group.
robots.txt controls crawling, not indexing. A blocked URL can still appear in search results if other pages link to it; use a noindex meta tag on a crawlable page to keep it out. The file must be served at the root of each host, for example https://www.example.com/robots.txt.

What does Robots.txt Generator do?

Free robots.txt generator with presets, including blocking AI training crawlers. Validate rules with an RFC 9309 parser and test paths before you upload.

Price
Free
Account
Not required
Processing
Entirely in your browser; your data and files are not uploaded
Works on
Any modern browser on desktop, tablet or phone
Category
SEO Tools

How to use the robots.txt generator

  1. Start from a preset: allow all, block all, block admin paths or block AI training crawlers.
  2. Edit each group: list the user-agents, then the paths to disallow and allow (one per line).
  3. Add your sitemap URLs.
  4. Read the validation notes and test a crawler token and path to confirm the result.
  5. Copy or download robots.txt and upload it to the root of your site.

Worked example

Blocking WordPress admin but allowing admin-ajax

With "Disallow: /wp-admin/" and "Allow: /wp-admin/admin-ajax.php" for all crawlers, testing Googlebot on /wp-admin/admin-ajax.php shows Allowed (the longer allow rule wins), while /wp-admin/options.php shows Blocked by "Disallow: /wp-admin/".

How it works

The file is generated from your groups and parsed back with our RFC 9309 implementation: user-agent matching is case-insensitive, the most specific matching group applies (combined if repeated), the longest matching rule wins, allow wins ties, * matches any characters and $ anchors the end. Paths are percent-encoding normalised before comparison.

Assumptions

  • The AI training preset uses crawler tokens verified from each operator’s documentation on 2026-09-25.
  • Crawl-delay is written if you enter it, but Google ignores it.

Frequently asked questions

Does robots.txt remove pages from Google?

No. It controls crawling, not indexing. A blocked page can still be indexed if other sites link to it. To keep a page out of results, allow crawling and add a noindex meta tag or header.

Will blocking AI crawlers affect my Google rankings?

Blocking Google-Extended or other AI training tokens does not affect Google Search. Blocking Googlebot would. The preset leaves search crawlers allowed.

Do all crawlers obey robots.txt?

No. It is a voluntary standard. Reputable crawlers follow it, but it cannot technically block access, and some operators say user-triggered fetchers may not follow it.

Where do I put the file?

At the root of each host, for example https://www.example.com/robots.txt. Subdomains need their own file.

Limitations

  • Validates syntax and matching logic only; it cannot tell which paths your site should block.
  • Non-standard directives are flagged rather than interpreted.