robots.txt Generator
Select crawler groups, add your sitemap URL, and copy a production-ready robots.txt file.
Presets
How it works
Simple crawler rules in three steps.
1. Select bots
Select search engine and AI crawler groups.
2. Add a sitemap
Point crawlers to your XML sitemap.
3. Copy the output
Save it as /robots.txt in your domain root.
Related Guides
Learn More
AI Bots and robots.txt
Learn the concepts behind this tool and improve your GEO setup.
llms.txt Guide
Learn the concepts behind this tool and improve your GEO setup.
Structured Data and JSON-LD
Learn the concepts behind this tool and improve your GEO setup.
FAQ
robots.txt Generator FAQ
Should I allow AI bots?
If your goal is AI search visibility, allowing relevant AI search crawlers can help discovery. Only allow bots you consider appropriate.
Where should robots.txt be uploaded?
Upload the file to your domain root, for example https://example.com/robots.txt.