How to see AI crawler visits in your server logs

Open your access log, nginx keeps it at /var/log/nginx/access.log and Apache at access_log, and grep case-insensitively for the crawler tokens. Then verify every hit against the vendor's published IP list, because user-agent strings are trivially spoofed. A token in a log line is a claim, not proof.

Why the visits are hard to read

1. Token names do not match product names. GPTBot is not called anything with ChatGPT in it, and Google-Extended never appears in logs even though it shows up in every robots.txt template.
2. The fetchers that ignore robots.txt still leave traces. ChatGPT-User and Perplexity-User show up in logs even when the file says nothing about them.
3. Scrapers borrow bot tokens. A grep hit without IP verification is unreliable; Microsoft's own guidance is never to judge authenticity from the user-agent alone.

Step 1: Find your logs

On a typical server: /var/log/nginx/access.log for nginx, access_log for Apache. On a managed host or a platform like Shopify, you may not have log access at all, so check with your host before going further. Without the log file, this guide stops here.

Step 2: Grep per token family, case-insensitive

Match the product token, not the full string:

grep -i "gptbot\|oai-searchbot\|chatgpt-user" /var/log/nginx/access.log
grep -i "claudebot\|claude-searchbot\|claude-user" /var/log/nginx/access.log
grep -i "perplexitybot\|perplexity-user" /var/log/nginx/access.log
grep -i "meta-external\|facebookexternalhit" /var/log/nginx/access.log
grep -i "bingbot" /var/log/nginx/access.log

Do not grep for Google-Extended: it is a policy token with no user-agent string, so any log line claiming it is spoofed. Do not grep for Claude-Web or anthropic-ai: those names are retired.

Step 3: Read a hit

A real hit shows a timestamp, a request for a real page on your site, a 200 status, and the token in the user-agent field. Read the token, not the brand: a Perplexity-User hit means someone asked Perplexity a question and it visited your page to answer, which is the stronger citation signal. A PerplexityBot hit means the indexer came by on its own schedule. No hits after a week of the file saying allow means something upstream is still blocking. One quirk: OpenAI's bots may append a robots.txt marker to the user-agent when fetching your robots.txt, so do not be surprised by a variant string there.

Step 4: Verify the hit is real

User-agent strings alone are spoofable. Verify the IP:

• OpenAI publishes per-bot IP ranges as JSON: openai.com/gptbot.json, openai.com/searchbot.json, and openai.com/chatgpt-user.json. Match the requesting IP against the file for that bot.
• Anthropic publishes https://claude.com/crawling/bots.json. It is not labeled per bot, so an IP match confirms Anthropic but cannot tell you which of the three bots visited.
• Perplexity publishes per-bot JSON: perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json.
• Meta publishes no crawler IP ranges. Verify via whois that the IP belongs to Meta's network block, AS32934.
• Googlebot is verified Google's way: reverse-DNS the IP, confirm the hostname resolves to googlebot.com, then forward-resolve it back to the same IP.

Step 5: Know what you will not find

You will not find Grok: xAI documents no crawler token and no IP ranges, and its fetches arrive as spoofed browsers from unrelated networks. You will not reliably find DeepSeek: it names no user-agent string, so a grep for DeepSeekBot matches third-party sightings, not anything official. Absence of an attributable hit proves nothing for either of them.

What this guide does not cover

Logs tell you who came. They do not tell you why a crawler that visited still does not cite you, and they cannot see the fetches that never identify themselves. Getting cited is the next job.