Aizii Learn · AI Search [← Back to Learn](https://aizii.ai/learn) # How to allow AI crawlers in robots.txt Add an Allow rule for each AI crawler token in \[yourdomain.com\]/robots.txt, replacing yourdomain.com with your actual domain, placed before any broad disallow rules. The block below covers OpenAI's three crawlers, Anthropic's three, Perplexity's two, Bing's bingbot, and Meta's crawler tokens. Allowing a crawler grants permission and nothing more: it does not change how you rank, it does not guarantee a crawl, and it has no effect on fetchers that ignore the file. ## Why the allow goes wrong 1. People allow the training bot and miss the search bot. GPTBot and OAI-SearchBot are different crawlers with different jobs, and most templates name only one. 2. Pasted blocks age badly. Claude-Web and anthropic-ai are retired names that match nothing now, and forum lists add tokens no vendor documents, like GrokBot and DeepSeekBot. 3. The file only governs crawlers that obey it. ChatGPT-User, Perplexity-User, and meta-externalfetcher are user-initiated and generally ignore robots.txt, so an allow block cannot control them. ## Step 1: Check the file exists Open \[yourdomain.com\]/robots.txt. If nothing loads, create the file at the root of your site. An allow block needs a file to live in. ## Step 2: Paste the block before any broad disallow rules ``` User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: bingbot Allow: / User-agent: meta-webindexer Allow: / User-agent: meta-externalagent Allow: / User-agent: meta-externalfetcher Allow: / User-agent: facebookexternalhit Allow: / User-agent: meta-externalads Allow: / ``` ## Step 3: Know what each allow does and does not do - OAI-SearchBot feeds ChatGPT's live search answers. Sites that opt out will not be shown in ChatGPT search answers. This is the one that matters this week. - GPTBot feeds training data for future models. Allowing it has no effect on search visibility; blocking it has no effect either. The settings are independent. - ChatGPT-User fetches pages when someone asks ChatGPT to check something. Because it is user-initiated, robots.txt rules may not apply to it. - Claude-SearchBot builds Anthropic's search index. Blocking it prevents search indexing. - ClaudeBot feeds training datasets only. Blocking it excludes future materials from training and nothing else. - Claude-User fetches pages in response to a user query. Blocking it stops Claude retrieving your content when asked. - PerplexityBot surfaces and links sites in Perplexity results. It is not used for training. It respects robots.txt. - Perplexity-User visits pages to answer a real user's question, and that visit is the stronger citation signal. It generally ignores robots.txt because a user requested the fetch. - bingbot is the index crawler behind Copilot. Microsoft publishes no AI-specific crawler token; AI visibility runs through the Bing index. - meta-webindexer is the key allow for Meta AI. Meta says allowing it helps Meta AI cite and link to your content. - meta-externalagent crawls to train Meta's foundation models. meta-externalads crawls for advertising products. meta-externalfetcher handles user-initiated fetches, including for AI agents, and may bypass robots.txt. facebookexternalhit fetches link previews. Do not add GrokBot or DeepSeekBot. Neither vendor documents a crawler token, so the rules match nothing, and both companies' live fetches bypass robots.txt anyway. One more: if your template includes User-agent: Google-Extended, know it is a policy token, not a crawler. It has no user-agent string and never appears in server logs. It controls whether already-crawled content may be used for Gemini training and grounding, and it has zero effect on Search inclusion, ranking, or AI Overviews. ## Step 4: Verify the change took Refetch \[yourdomain.com\]/robots.txt and confirm the block is live. Then watch your server logs over the next day or two: OpenAI, Perplexity, and Meta all say robots.txt changes take about 24 hours to take effect. ### The fundamentals These mechanics are shared across every engine and covered in full in their own guides: - •The complete rule reference for every crawler and edge case, see our [robots.txt guide for AI search](https://aizii.ai/learn/ai-search/how-to-write-robots-txt-for-ai-search) - •What goes in llms.txt and how to publish it, see our [guide to writing a good llms.txt file](https://aizii.ai/learn/ai-search/how-to-write-a-good-llms-txt-file) - •The markup AI engines read to verify price and availability, see our [product schema guide for AI search](https://aizii.ai/learn/ai-search/how-to-add-product-schema-markup-for-ai-search) ### Say you run a skincare brand You paste an allow block from a forum. It covers GPTBot but not OAI-SearchBot, so ChatGPT's training pipeline has permission while its live search does not. It also includes Claude-Web, a retired name that matches nothing, and GrokBot, which no vendor documents. A month later ChatGPT still cites a competitor in live answers. The file looked complete. It allowed the wrong bots. ## What this guide does not cover This is the permission slip. It does not describe your products to the models, it does not get you crawled faster, and it does not cover the edge settings that can overrule the file after you save it. ## Frequently Asked Questions Can I use wildcard user-agent rules to allow AI bots while disallowing other scrapers? Yes, but you must place specific User-agent blocks above any generic wildcard disallow. Rules in robots.txt are evaluated by matching the most specific user-agent token. Does allowing a crawler in robots.txt guarantee it will visit my site? No. Robots.txt grants permission, but crawlers allocate visit budgets based on site authority, sitemap discovery, and overall link structure. Should I add Crawl-delay for AI bots? Avoid adding crawl delays unless your server is experiencing performance degradation. High crawl delays often cause AI crawlers to abort scheduled refreshes. ## Check your own site now Run the free scan. One click, no signup, and it shows which AI crawlers your robots.txt actually allows: aizii.ai/aeo-scan [Run the free scan](https://aizii.ai/aeo-audit)