robots.txt Generator
Select crawler groups, add a sitemap URL and copy a clean starting point for your domain.
Preset
How it works
Simple crawler rules in three steps.
1. Choose bots
Select search and AI crawler user-agents.
2. Add sitemap
Point crawlers to your XML sitemap.
3. Copy output
Save it as /robots.txt in your root directory.
Related guides
Learn more
AI Bots & robots.txt
Learn the concept behind this tool and improve your AI visibility setup.
llms.txt Guide
Learn the concept behind this tool and improve your AI visibility setup.
Schema.org & JSON-LD
Learn the concept behind this tool and improve your AI visibility setup.
FAQ
robots.txt Generator FAQ
Should I allow AI bots?
If your goal is AI search visibility, allowing relevant AI search crawlers can help discovery. Only allow bots you are comfortable with.
Where should robots.txt be uploaded?
Upload it to the root of your domain, for example https://example.com/robots.txt.