Why your robots.txt says allow but AI crawlers still can't get in

Your robots.txt allows the AI crawlers and they still never show up. The file is not the problem. A robots.txt file is a declaration; it is not enforcement. Between the crawler and your site sits your edge: Cloudflare, an AWS WAF, or a managed edge platform your agency runs. If any of those blocks the crawler's user-agent or IP range, the request dies before your server ever sees it, and the allow rule in your file never gets a vote.

Why the file and the edge disagree

Three things commonly stand in front of your site and make their own decisions:

1. Cloudflare bot management. Bot Fight Mode and the bot rules under Security > Bots challenge or block traffic Cloudflare classifies as automated. AI crawlers are automated by definition, so a rule meant to stop scrapers stops GPTBot and PerplexityBot with the same hand. This is the single most common cause our scans surface.
2. AWS WAF bot-control rules. The AWS managed bot-control rule group scores requests and blocks the ones that look like bots. AI crawler traffic matches the pattern the rule was built to catch. Blocking can happen by user-agent token or by IP reputation, and it happens before the request reaches your origin.
3. Managed vendor edges. Some agencies and platforms run a shared edge in front of client sites, the SourceFlow-type setups, with its own bot policy. Your robots.txt can be perfect while the vendor's edge quietly drops every AI crawler at the door. If you do not control the edge, you do not control crawler access.

Note what makes this confusing: the blockers do not read your robots.txt. A WAF does not check whether you allowed the crawler before deciding to block it. The file governs polite crawlers that knock; the edge governs all traffic that arrives. When the two disagree, the edge wins every time.

Fix it in this order

1. Confirm what the file actually says

Open [yourdomain.com]/robots.txt in a browser, replacing yourdomain.com with your actual domain. Verify the AI crawler tokens are allowed with their own rules, placed before any broad disallow. A wildcard disallow above the allow rules can shadow them depending on how the crawler parses the file. Check for typos in the token names: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, meta-webindexer, bingbot. If the file is correct, the problem is upstream of it, and the next steps prove that.

2. Test from outside with a real bot user-agent

Run this from any machine outside your network, replacing yourdomain.com with your actual domain:

curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" https://[yourdomain.com]/robots.txt

Then run the same request with a normal browser user-agent and compare. If the browser gets a 200 and the bot user-agent gets a 403, a 429, or a challenge page, the edge is blocking declared crawlers regardless of what the file says. Try it against a product page too, not just robots.txt: some setups allow the file itself but block the crawler on content pages.

3. Check the WAF and bot-management settings

In Cloudflare, open Security > Bots and look for anything challenging or blocking automated traffic: Bot Fight Mode, or rules set to block "definitely automated" requests. These do not distinguish a search crawler from a scraper unless you tell them to. Add explicit allowances for the AI crawlers you want, and if you block by IP, allowlist the ranges the vendors publish: OpenAI publishes per-bot IP ranges as JSON files, Anthropic publishes its crawler prefixes, and Perplexity publishes its bot IP lists. Perplexity's own docs include Cloudflare and AWS WAF allowlist guidance, which is worth following verbatim since it comes from the crawler's operator.

In AWS, open the WAF console and inspect the bot-control managed rule group for blocks on crawler user-agents. If a managed edge platform sits in front of your site, ask the vendor directly whether their edge filters AI crawler traffic and how to allowlist it. Do not assume they mirror your robots.txt; they almost certainly do not.

4. Confirm in your server logs whether the crawler ever arrives

The logs are the ground truth. Grep for the crawler tokens:

grep -i "gptbot\|oai-searchbot\|perplexitybot\|claude-searchbot\|meta-webindexer\|bingbot" /var/log/nginx/access.log

If the file allows them, the edge allows them, and the logs still show no hits after a day or two, something further upstream is still filtering, or the crawlers simply have not come back yet. If the logs show hits arriving with 200 responses, the path is clear and the problem was never access. No hits plus a clean curl test from step 2 means the block is still in place somewhere you have not looked.

What this guide does not cover

Opening the edge does not help with engines that ignore the file entirely. Grok's answer-time fetches arrive as spoofed browsers through proxy networks, so no robots.txt rule and no allowlist governs them; that is a separate problem. This guide also does not cover monitoring: crawler behavior and WAF rules drift, and a setup that works today can silently close again after a vendor update. And access is not recommendation: once crawlers can reach your pages, they still need verifiable facts to cite, which is the product schema guide's job.