Free tool

AI Crawler robots.txt Generator

Control exactly which AI crawlers can read your site. Toggle each bot on or off and generate a clean robots.txt you can paste straight onto your domain.

Crawler access

robots.txt

robots.txt
# Standard crawlers
User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: CCBot
Allow: /

User-agent: Bytespider
Allow: /

User-agent: meta-externalagent
Allow: /

How to use it

  1. 1Toggle each AI crawler to allow or block it.
  2. 2Optionally add your sitemap URL.
  3. 3Copy or download the generated robots.txt.
  4. 4Upload it to the root of your domain (yoursite.com/robots.txt).

Frequently asked

Should I block AI crawlers?

If you want to appear in AI answers, allow them. Blocking AI crawlers like GPTBot or PerplexityBot means your content can't be read or cited by those engines.

What is Google-Extended?

Google-Extended controls whether your content is used to train and ground Google's AI products. Allowing it keeps you eligible for Google's AI features.

Will robots.txt remove me from AI answers instantly?

Crawlers re-read robots.txt on their own schedule, so changes take effect the next time each bot visits, not immediately.

Want this on autopilot?

Rankvolt researches, writes, and publishes citation-ready articles daily — so you become the answer AI recommends and rank on Google.

See how Rankvolt worksBrowse all tools