Aizii Learn · AI Search

[← Back to Learn](https://aizii.ai/learn)

# How to write robots.txt for AI search

robots.txt is the one file every AI crawler checks before it touches your site. It controls which bots may crawl your pages, and nothing else. It does not control what models already know about you, it does not reliably stop answer-time fetches, and it is only half the story: the file declares your rules, but your firewall and CDN decide what actually gets through.

There are two classic mistakes. The first is blocking more than you intend: blocking a training bot while trying to control search, or blocking everything with a broad rule you forgot about. The second is getting the file right while a bot manager at the edge blocks the crawlers anyway. This guide covers every AI crawler token, what blocking each one actually costs, and how to verify the whole thing end to end.

## What robots.txt does and does not control

It controls automated crawling: which bots may request your pages and which paths are off limits to them. For AI search, that means it controls which index crawlers and training crawlers can read your site.

It does not control what a model already knows. If your content is already in training data or an index, a robots.txt change does not remove it.

It does not reliably control answer-time fetches. When a user asks ChatGPT or Perplexity a question and the tool fetches your page to answer, several of those fetchers ignore robots.txt or may bypass it. The reference below marks them.

It does not control the edge. Cloudflare, AWS WAF, and other bot managers enforce their own rules before your file is ever read. A file that says allow means nothing if the edge blocks the bot.

## Every AI crawler token

Match on the token, not the full user-agent string. Tokens are the stable part; version numbers and URL suffixes change.

### OpenAI

OAI-SearchBot is the search retrieval crawler. OpenAI's wording: it is used to surface websites in search results in ChatGPT's search features, and opted-out sites will not be shown in ChatGPT search answers. Block it and you disappear from ChatGPT search answers. This is the one that matters for being recommended this week.

GPTBot is the training crawler. It crawls content that may be used to train future foundation models. Block it and you opt out of training only. OpenAI's own docs say the settings are independent, so blocking GPTBot has no effect on search visibility.

ChatGPT-User is the per-request fetcher for user actions, including GPT actions. It is not a background crawler. Block it and ChatGPT can never check your live pages while answering, so it cannot verify your current prices or stock. Know its limit: because these fetches are user-initiated, robots.txt rules may not apply to it.

OAI-AdsBot validates pages submitted as ChatGPT ads, and its data is not used for training. Rarely relevant unless you run ChatGPT ads.

Full user-agent strings look like `...compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot`. OpenAI publishes per-bot IP ranges (openai.com/searchbot.json, gptbot.json, chatgpt-user.json, adsbot.json), and its bots may append a robots.txt marker to the user-agent when fetching the file itself. Expect changes to take about 24 hours to take effect.

### Anthropic

ClaudeBot is the training crawler. Blocking it excludes your content from future training datasets only. Nothing changes for search.

Claude-SearchBot is the search index crawler. Anthropic's wording: it navigates the web to improve search result quality. Block it and you are not indexed for Claude search.

Claude-User is the answer-time fetcher. When someone asks Claude a question, it may retrieve your page to build the answer. Block it and Claude cannot pull your content into an answer.

Match on the token. No full user-agent string for these has been verified from a primary source. The old names Claude-Web and anthropic-ai are retired; do not grep for those. Anthropic honors robots.txt for all three bots, supports a non-standard Crawl-delay, and asks site owners not to IP-block its crawlers, because that impedes its robots.txt reads. Opt-outs are per subdomain.

### Perplexity

PerplexityBot surfaces and links websites in Perplexity search results. Perplexity's docs say it is not used to crawl content for foundation models, and it respects robots.txt. Block it and you do not surface or get linked in Perplexity results.

Perplexity-User fetches a page to answer a specific user question. It is not used for crawling or training. There is nothing to control here: since the fetch is user-requested, this fetcher generally ignores robots.txt rules.

Full user-agent strings, verbatim from Perplexity's docs:

`Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)`

`Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)`

Changes take up to 24 hours. One caveat: in August 2025 Cloudflare reported Perplexity using undeclared crawlers to reach sites that had blocked its declared bots, and removed it from its verified-bot list. Perplexity disputed the report. Both sides stated their case. The practical read: declared-agent rules alone may not equal full exclusion.

### Microsoft (Bing and Copilot)

There is no AI-specific crawler token from Microsoft. The index crawler is bingbot, with the classic user-agent `compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm`. AI visibility on the Microsoft side is controlled by meta directives, not a bot token. Block bingbot and you lose the Bing index, which feeds Copilot and is one of ChatGPT's retrieval sources.

### Meta

Per Meta's crawler documentation, meta-externalagent/1.1 crawls to train foundation AI models or improve products by indexing content directly. Blocking it opts you out of training. meta-webindexer/1.1 exists to improve Meta AI search results; Meta says allowing it helps Meta AI cite and link to your content, which makes it the key allowlist for citations. meta-externalads/1.1 crawls for advertising and business products. meta-externalfetcher/1.1 fetches individual links when a user requests them, including for AI agents, and Meta says it may bypass robots.txt. facebookexternalhit/1.1 is the link-preview fetcher for shared URLs and may bypass robots.txt during security checks.

The full form looks like `meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)`; the short token form is also accepted. Changes take up to 24 hours.

Sourcing note: we could not load Meta's crawler docs page directly during our research, so treat this token list as strongly transcribed rather than confirmed. The details are consistent across seven independent transcriptions with zero contradictions.

### The policy token that is not a crawler

Google-Extended is a robots.txt policy token, not a crawler. It has no HTTP user-agent string and will never appear in your server logs. Any log line claiming it is spoofed.

It controls whether content Google has already crawled may be used to train future Gemini generations and for grounding, which is handing Google Search index content to the model at prompt time. It has zero effect on Search inclusion, ranking, or AI Overviews. Write it as a group in the file:

```
User-agent: Google-Extended
Disallow: /
```

Verify it the same way: fetch your robots.txt and confirm the group is there. There is no log check.

### Tokens with no vendor behind them

GrokBot and DeepSeekBot appear in third-party bot directories and log roundups, but neither xAI nor DeepSeek publishes a crawler user-agent, IP ranges, or robots.txt guidance. We checked both vendors' documentation directly and found nothing. An allow rule for either token is best effort. It may catch some crawler traffic, but it will not gate answer-time fetches, which arrive as ordinary browsers. Treat any guide that presents these strings as official with suspicion: it is guessing.

One related gap: Meta's personal agent, Muse, has no named crawler token at all. It browses from its own VM with a standard browser user-agent. There is no allowlist lever to pull for it. The levers are renderability and structured data.

## A starting allowlist for AI visibility

If your goal is to be found and cited by AI tools, these are the retrieval bots to allow. Training bots are a separate decision each brand makes for itself.

```
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: meta-webindexer
Allow: /

User-agent: bingbot
Allow: /
```

Put these groups near the top of the file, before any broad disallow rules. GPTBot, ClaudeBot, and meta-externalagent are the training crawlers: allowing them lets the vendors train on your content, blocking them opts you out of training, and neither choice affects whether you appear in answers.

Rather than assembling this by hand, run the free scan at aizii.ai/aeo-scan and hit fixes. Enter your email and it generates these rules from your actual site.

## The file declares, the edge enforces

This is where most robots.txt work silently fails. Your file can allow every bot on this page while Cloudflare Bot Fight Mode or an AWS WAF rule blocks them at the edge. The check is simple: after setting the file, look at your own logs.

```
grep -i "oai-searchbot\|claude-searchbot\|perplexitybot\|meta-webindexer\|bingbot" /var/log/nginx/access.log
```

No hits after a week or two means something upstream is still blocking, regardless of what the file says. In Cloudflare, open Security, then Bots, and confirm AI crawlers are not being challenged or blocked.

Authenticate what you find. OpenAI publishes per-bot IP ranges you can check a request against. Anthropic publishes its crawler prefixes at claude.com/crawling/bots.json. Meta publishes no IP ranges; verify Meta traffic with a whois lookup against its network block, AS32934. For bingbot, Microsoft publishes an official IP list and says never to judge authenticity from the user-agent alone.

## Editing the file on your platform

The rules are the same everywhere, but where you edit the file depends on your platform. See the platform guides:

-   [How to edit robots.txt on Shopify for AI crawlers](https://aizii.ai/learn/ai-search/how-to-edit-robots-txt-on-shopify-for-ai-crawlers)
-   [How to edit robots.txt on WooCommerce for AI crawlers](https://aizii.ai/learn/ai-search/how-to-edit-robots-txt-on-woocommerce-for-ai-crawlers)
-   [How to edit robots.txt on BigCommerce for AI crawlers](https://aizii.ai/learn/ai-search/how-to-edit-robots-txt-on-bigcommerce-for-ai-crawlers)
-   [How to edit robots.txt on Salesforce Commerce Cloud for AI crawlers](https://aizii.ai/learn/ai-search/how-to-edit-robots-txt-on-salesforce-commerce-cloud-for-ai-crawlers)
-   [How to edit robots.txt on Adobe Commerce for AI crawlers](https://aizii.ai/learn/ai-search/how-to-edit-robots-txt-on-adobe-commerce-for-ai-crawlers)
-   [How to edit robots.txt on Wix for AI crawlers](https://aizii.ai/learn/ai-search/how-to-edit-robots-txt-on-wix-for-ai-crawlers)
-   [How to edit robots.txt on Squarespace for AI crawlers](https://aizii.ai/learn/ai-search/how-to-edit-robots-txt-on-squarespace-for-ai-crawlers)
-   [How to edit robots.txt on a self-hosted site for AI crawlers](https://aizii.ai/learn/ai-search/how-to-edit-robots-txt-on-a-self-hosted-site-for-ai-crawlers)
-   [How to edit robots.txt on a custom website for AI crawlers](https://aizii.ai/learn/ai-search/how-to-edit-robots-txt-on-a-custom-website-for-ai-crawlers)

## How to test the whole thing

1.  Fetch \[yourdomain.com\]/robots.txt in a browser, replacing yourdomain.com with your actual domain. Confirm your groups are present and ordered before any broad disallow.
2.  Request your homepage as a bot and confirm the edge lets it through: `curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" -I https://[yourdomain.com]/` should return a 200, not a block page.
3.  Grep your logs for the tokens and validate the IPs against the published ranges.
4.  Wait. OpenAI, Perplexity, and Meta all take up to about 24 hours to pick up robots.txt changes.

## Common mistakes

Blocking a training bot to control search. GPTBot and ClaudeBot have no effect on whether you appear in answers. If you want out of ChatGPT answers, the token is OAI-SearchBot. If you want out of training, it is GPTBot. They are independent settings.

Treating Google-Extended as a crawler. It never appears in logs, and adding it to a log-based blocklist does nothing.

Relying on robots.txt for answer-time fetches. ChatGPT-User, Perplexity-User, meta-externalfetcher, and Muse's browser either ignore the file or may bypass it. For these, legible HTML and structured data are the levers, not the file.

Assuming the file equals enforcement. Check the edge. A file that says allow while the firewall says block is the failure this guide exists to prevent.

Forgetting propagation. Test the day after you change the file, not ten minutes after.

## What this guide does not cover

Robots.txt gets crawlers to your door. It does not make your pages legible once they arrive, it does not keep your facts consistent across the web, and it does not let an agent complete a purchase. For the map you hand the models, see our [guide to writing a good llms.txt file](https://aizii.ai/learn/ai-search/how-to-write-a-good-llms-txt-file). For the ChatGPT-specific playbook, see our guide to getting ChatGPT to recommend your brand.

## Frequently Asked Questions

What is the recommended rule order in robots.txt for AI search bots?

Place specific User-agent declarations for bots like OAI-SearchBot, Claude-SearchBot, and PerplexityBot at the top, followed by general search bots and lastly wildcard rules.

Can I disallow training crawlers while allowing search crawlers?

Yes. You can disallow GPTBot and ClaudeBot while explicitly allowing OAI-SearchBot and Claude-SearchBot to participate in live search answers without training.

How large can a robots.txt file be before crawlers truncate it?

Most search engines and AI crawlers truncate robots.txt files exceeding 500 kilobytes. Keep your file concise and focused.

## Check your own brand now

Run the free scan. One click, no signup, and it shows exactly which AI crawlers can reach your site today: aizii.ai/aeo-scan

[Run the free scan](https://aizii.ai/aeo-audit)