How to check if your site is blocking AI crawlers

Pull [yourdomain.com]/robots.txt, replacing yourdomain.com with your actual domain, and read every AI crawler group in it. Then test the live site with a bot user-agent from outside your network, check your edge and bot settings, grep your server logs, and confirm in webmaster tools. The file is only step one: the file declares, the edge enforces, and some of the fetchers that matter ignore the file entirely.

Why the block hides

1. robots.txt files are written once and forgotten. A wildcard disallow added to hide a staging site or fight scrapers quietly covers every AI crawler too.
2. The declaration is not the enforcement. The file can say allow while Cloudflare Bot Fight Mode or an AWS WAF rule blocks the same bots at the edge.
3. Some of the fetchers that matter most, ChatGPT-User, Perplexity-User, and meta-externalfetcher, generally ignore robots.txt because a user requested the fetch. The file can be perfect and still not tell the whole story.
4. Token names mislead. Google-Extended shows up in every template as if it were a crawler, but it is a policy token with no user-agent string. It will never appear in your logs; any log line claiming it is spoofed.

Step 1: Fetch and read your robots.txt

Open [yourdomain.com]/robots.txt in a browser. If nothing loads, create the file at the root of your site before doing anything else. Read every group:

• A User-agent: * block with Disallow: / blocks every crawler, including the AI ones, unless a more specific group overrides it.
• Look for the AI tokens by name: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, bingbot, and Meta's meta-externalagent, meta-webindexer, meta-externalfetcher, and facebookexternalhit.
• Token-specific groups belong before any broad disallow rules.

Step 2: Test from the outside with a bot user-agent

Your server can serve one thing to browsers and another to bots. Test it from outside your network, your laptop works, using a user-agent string built around the documented token. The token is the part every bot rule keys on:

curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)" https://[yourdomain.com]/

curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://[yourdomain.com]/

curl -s -o /dev/null -w "%{http_code}\n" -A "ClaudeBot" https://[yourdomain.com]/

Anthropic publishes no official full user-agent string, so for ClaudeBot the token alone is the reliable test. Run the same request without the -A flag as a normal browser. If the browser gets a 200 and the bot gets a 403, a challenge page, or a redirect, your server or edge treats the crawler differently, whatever the file says.

Step 3: Check the edge, not just the file

This is the finding that turns up most often. In Cloudflare, open Security > Bots and confirm AI crawlers are not being challenged or blocked, and check Bot Fight Mode. In AWS, review your WAF bot-control rules. The robots.txt file is what you declare; the edge is what enforces.

Step 4: Grep your server logs

The logs are the ground truth. On a typical server they live at /var/log/nginx/access.log for nginx or access_log for Apache. On a managed host, ask your host where to find them.

grep -i "gptbot\|oai-searchbot\|chatgpt-user\|claudebot\|claude-searchbot\|claude-user" /var/log/nginx/access.log
grep -i "perplexitybot\|perplexity-user\|meta-external\|facebookexternalhit\|bingbot" /var/log/nginx/access.log

Match the token, not the full string. No hits after a week of the file saying allow means something upstream is still blocking.

Step 5: Confirm in webmaster tools

Bing Webmaster Tools has a robots.txt tester and a "Fetch as Bingbot" check. Use both, since Copilot retrieves from Bing's index. In Google Search Console, use URL Inspection's live test and Crawl Stats to confirm Googlebot reaches your pages.

What this guide does not cover

This guide finds the block. It does not write the allow rules, and it does not cover what comes after: when nothing is blocked and you still do not show up, the problem is indexing, verification, or the freshness of the underlying index, which are separate jobs.