robots.txt: Set crawler access deliberately
Separate crawl policy from content publishing, authentication and AI measurement.
Reviewed against Optimly’s implementation on .
What robots.txt does
The Robots Exclusion Protocol asks participating crawlers which paths they may fetch. It does not secure private content, erase training data or guarantee indexing. Protect private resources with authentication and authorization. See RFC 9309.
Rules in a matching user-agent group apply to that crawler; the wildcard group is the fallback when there is no match. Do not assume a bot-specific group inherits your wildcard restrictions.
What Optimly serves
The current public index allows the named crawler groups and the wildcard group, and advertises its sitemap. Comments point to the machine-readable resources. This is Optimly’s public-index policy, not a required policy for your website.
A minimal example for your own site
Example only: adapt paths and the sitemap URL to your website. These restrictions are crawl requests, not protection for account or administration pages.
User-agent: *
Disallow: /account/
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlIf you add named crawler groups, put the restrictions you intend in those groups too. Test the resulting policy against the crawler’s documentation.
Search, training and user-requested visits differ
OpenAI distinguishes OAI-SearchBot for search, GPTBot for potential training use, and ChatGPT-User for user-triggered visits. Its documentation notes that robots.txt rules may not apply to those user-triggered visits. Choose policies by purpose and consult each provider’s current guidance. OpenAI crawler documentation →
A blocked crawl does not prove a model has forgotten a brand. An allowed crawl does not prove that its content was used in an answer.
Check the result
- Fetch /robots.txt and confirm a successful plain-text response.
- Check the applicable group and each important public path.
- Check that your CDN or firewall is not separately blocking desired requests.
- Inspect server logs for successful fetches; logs establish requests, not model training or recommendations.
- Keep your sitemap and public links current.
Keep your published record useful
Review your brand information, its sources and its dates. Check the public record after publishing, and use reports to understand the answers actually observed.