Categories
Most popular
PDF
Images
Video and audio
US mortgages
Australian home loans
Loans and savings
Everyday and business
Documents
Pricing and profit
Marketing and SEO
Marketing guides
CRM and automation
Latest articles
Free CRM templates
Download, edit and import into the tool you use. No email required.
Search engine optimisation, structured data and AI crawlers.
How robots.txt works, its syntax and limits, how to test it, and the documented user agents for AI crawlers such as GPTBot, ClaudeBot and Google-Extended.
No articles match your search.
Accessibility
Settings are saved in this browser only.