Loading tool...
Loading tool...
Build a robots.txt file visually. Control crawler access per bot, add global path rules, and include your sitemap URL.
No global path rules — all paths are allowed by default.
User-agent: * Allow: /
2 lines · 22 bytes
About robots.txt
The robots.txt file lives at your website root (e.g. https://example.com/robots.txt) and tells well-behaved crawlers which pages to visit. It is not a security mechanism — malicious bots can ignore it.
AI training bots like GPTBot and ClaudeBot respect Disallow rules per their policies.