Aizii Learn · AI Search

[← Back to Learn](https://aizii.ai/learn)

# How to see AI crawler visits in your server logs

Open your access log, nginx keeps it at /var/log/nginx/access.log and Apache at access\_log, and grep case-insensitively for the crawler tokens. Then verify every hit against the vendor's published IP list, because user-agent strings are trivially spoofed. A token in a log line is a claim, not proof.

## Why the visits are hard to read

1.  Token names do not match product names. GPTBot is not called anything with ChatGPT in it, and Google-Extended never appears in logs even though it shows up in every robots.txt template.
2.  The fetchers that ignore robots.txt still leave traces. ChatGPT-User and Perplexity-User show up in logs even when the file says nothing about them.
3.  Scrapers borrow bot tokens. A grep hit without IP verification is unreliable; Microsoft's own guidance is never to judge authenticity from the user-agent alone.

## Step 1: Find your logs

On a typical server: /var/log/nginx/access.log for nginx, access\_log for Apache. On a managed host or a platform like Shopify, you may not have log access at all, so check with your host before going further. Without the log file, this guide stops here.

## Step 2: Grep per token family, case-insensitive

Match the product token, not the full string:

```
grep -i "gptbot\|oai-searchbot\|chatgpt-user" /var/log/nginx/access.log
grep -i "claudebot\|claude-searchbot\|claude-user" /var/log/nginx/access.log
grep -i "perplexitybot\|perplexity-user" /var/log/nginx/access.log
grep -i "meta-external\|facebookexternalhit" /var/log/nginx/access.log
grep -i "bingbot" /var/log/nginx/access.log
```

Do not grep for Google-Extended: it is a policy token with no user-agent string, so any log line claiming it is spoofed. Do not grep for Claude-Web or anthropic-ai: those names are retired.

## Step 3: Read a hit

A real hit shows a timestamp, a request for a real page on your site, a 200 status, and the token in the user-agent field. Read the token, not the brand: a Perplexity-User hit means someone asked Perplexity a question and it visited your page to answer, which is the stronger citation signal. A PerplexityBot hit means the indexer came by on its own schedule. No hits after a week of the file saying allow means something upstream is still blocking. One quirk: OpenAI's bots may append a robots.txt marker to the user-agent when fetching your robots.txt, so do not be surprised by a variant string there.

## Step 4: Verify the hit is real

User-agent strings alone are spoofable. Verify the IP:

-   OpenAI publishes per-bot IP ranges as JSON: openai.com/gptbot.json, openai.com/searchbot.json, and openai.com/chatgpt-user.json. Match the requesting IP against the file for that bot.
-   Anthropic publishes https://claude.com/crawling/bots.json. It is not labeled per bot, so an IP match confirms Anthropic but cannot tell you which of the three bots visited.
-   Perplexity publishes per-bot JSON: perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json.
-   Meta publishes no crawler IP ranges. Verify via whois that the IP belongs to Meta's network block, AS32934.
-   Googlebot is verified Google's way: reverse-DNS the IP, confirm the hostname resolves to googlebot.com, then forward-resolve it back to the same IP.

## Step 5: Know what you will not find

You will not find Grok: xAI documents no crawler token and no IP ranges, and its fetches arrive as spoofed browsers from unrelated networks. You will not reliably find DeepSeek: it names no user-agent string, so a grep for DeepSeekBot matches third-party sightings, not anything official. Absence of an attributable hit proves nothing for either of them.

### The fundamentals

These mechanics are shared across every engine and covered in full in their own guides:

-   •The allow rules that create the visits you are looking for, see our guide to allowing AI crawlers in robots.txt
-   •The full five-step blocking diagnostic, see our [guide to checking whether your site blocks AI crawlers](https://aizii.ai/learn/ai-search/how-to-check-if-your-site-is-blocking-ai-crawlers)
-   •Every AI crawler token in one reference table, see our [robots.txt guide for AI search](https://aizii.ai/learn/ai-search/how-to-write-robots-txt-for-ai-search)

### Say you run an independent bookstore

You grep a month of logs. PerplexityBot visited your bestsellers page several times, each from an IP inside Perplexity's published range: real indexer visits. One line shows Google-Extended as the user-agent: spoofed, since that token has no user-agent string. And there is a cluster of ChatGPT-User hits on a single product page from the same afternoon: someone asked ChatGPT about that book, and it came to check. The logs do not tell you what the answer said, but they tell you which engines are paying attention.

## What this guide does not cover

Logs tell you who came. They do not tell you why a crawler that visited still does not cite you, and they cannot see the fetches that never identify themselves. Getting cited is the next job.

## Frequently Asked Questions

Where are crawler visits recorded on typical Linux web servers?

Visits appear in your HTTP access logs, typically located at /var/log/nginx/access.log or /var/log/apache2/access.log.

How can I filter my logs specifically for AI bot requests?

Use grep with regular expressions matching common tokens, such as grep -Ei 'gptbot|oai-searchbot|claudebot|perplexitybot' /var/log/nginx/access.log.

How can I verify that a crawler request is authentic and not spoofed?

Perform a reverse DNS lookup on the client IP address or compare it against the officially published IP ranges provided in JSON format by OpenAI, Anthropic, and Google.

## Check your own site now

Run the free scan. One click, no signup, and it shows which AI crawlers have actually been reaching your site: aizii.ai/aeo-scan

[Run the free scan](https://aizii.ai/aeo-audit)