How to get Grok to recommend your brand

Grok recommends brands it can read live on the open web and weigh against what is being said on X right now. xAI publishes no official crawler documentation, so there is no documented token, no IP ranges, and no webmaster console. And when Grok answers a question, its fetches often arrive as ordinary browsers that never request robots.txt. The playbook for Grok is therefore unlike the others: the allowlist is best-effort, clean readable HTML is the real lever, and posting on X does not get you recommended to buyers.

Why Grok skips your brand

Grok's failure modes are its own:

1. Answer-time fetches bypass robots.txt. Independent tests have watched Grok read robots.txt-blocked pages and arrive as spoofed browsers that never request the file. The declared rule is necessary, but it reaches only the crawl lane.
2. JavaScript-only content is invisible. Grok reads the page as plain HTML. If your price and specs appear only after scripts run, there is nothing to cite.
3. The "just post on X" consensus is wrong. CueScout measured 740 citations across 36 questions on three engines and found zero X citations on buying questions. X shapes real-time and sentiment questions, not buyer recommendations.
4. You cannot verify inclusion through any official channel. No crawler docs, no IP list, no webmaster tool. The only check is a honeypot canary URL, and even a hit cannot be attributed to Grok with confidence, because the traffic arrives as ordinary browsers.

Step 1: Allow GrokBot in robots.txt, and understand what it does not do

First, check your current robots.txt. Open [yourdomain.com]/robots.txt in your browser, replacing yourdomain.com with your actual domain. If nothing loads, you do not have one yet. Create it: a plain text file named robots.txt at the root of your site.

Then add this allow rule near the top, before any broad disallow rules. GrokBot is a real crawler: it shows up in server logs in the wild, so this rule catches genuine Grok crawl traffic. One honest note: xAI publishes no official crawler documentation, the way OpenAI and Google do, so this is the observed name rather than a documented one. Add it anyway. It costs nothing.

User-agent: GrokBot
Allow: /

Reload the file and confirm the rule is there.

Now the part most guides get wrong. The robots.txt rule above is necessary, but for Grok it is not sufficient. When Grok answers a user's question, its fetch traffic often arrives disguised as an ordinary browser, sometimes through proxy networks, and never requests robots.txt at all. A robots.txt rule cannot reach traffic that never reads the file.

Two independent tests documented this. SurfacedBy, running three rounds around September 2026, found Grok reading a robots.txt-blocked page all three rounds while never once requesting robots.txt. Fetches arrived as headless Chrome from a single Google Cloud address and as bursts from about 22 IPs across 10+ networks and six countries, all with spoofed browser user agents; none of the traffic registered to xAI, and the proxy traffic was attributed to a commercial proxy service. Separately, searchengineworld ran canary URLs across 12 AI assistants around July 2026 and found Grok presenting faked Mac-browser user agents from 17 distinct IPs, again not respecting robots.txt.

So do Step 1, but do not stop there: the pages themselves have to be legible as clean HTML (Step 2), because Grok reads whatever it fetches however it arrives.

One more check most guides skip: robots.txt is what you declare, your edge is what enforces. Cloudflare Bot Fight Mode, AWS WAF bot rules, and similar protections often block AI crawlers at the edge even when robots.txt allows them. If you run Cloudflare, check Security > Bots and make sure AI crawlers are not being challenged or blocked. The scan tests this directly by fetching your site as each crawler and comparing what comes back. A mismatch between what robots.txt says and what the edge does is the single most urgent finding in our scans.

Check for the observed token in your server logs:

grep -i "grokbot" /var/log/nginx/access.log

Do not treat a clean result as proof Grok never visits: some of its retrieval traffic arrives with ordinary browser user agents, indistinguishable from human visitors. The log check confirms the crawler lane is open, not that every Grok visit is counted.

While you're in there: add the other major AI crawlers too. You are already editing the file.

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

The complete reference for every AI crawler lives in our robots.txt guide for AI search.

Step 2: Make your pages legible as plain HTML

This is the real lever for Grok and the step the generic checklists skip. View source on a product page: if the price, description, and specs are in the HTML, you are fine. If they only appear after JavaScript runs, Grok's answer-time fetch reads an empty page, and robots.txt cannot help because the fetch never consulted it.

• Price, description, and specs as text in the HTML, never trapped behind scripts or inside images.
• Stable product URLs. Frequently changing URLs fragment whatever index presence you build.
• One clear H1 per page, with headings that match the questions buyers ask.

You do not need to rebuild the site. Most platforms can serve pre-rendered HTML without changing what shoppers see. This matters doubly for Grok's Deep Research mode: one site's published 2025 logs showed Grok arriving as a plain iPhone Safari user agent, never touching robots.txt, and re-fetching the same URL seven times from different IPs. The page Grok lands on has to carry the facts in the HTML itself, because that page is what gets read.

Step 3: Build a real X presence, honestly

This is the step every other Grok guide gets wrong. The consensus advice is "post on X and Grok will recommend you." The measured truth, from CueScout, who ran 740 citations across 36 questions on three engines: for buying questions ("best trail running shoes under $150"), X gets zero citations. Zero.

That does not mean X is worthless. It means X works for the questions it actually shapes: real-time topics (what happened today in your category), sentiment-shaped questions (what do people think of your brand right now), and brand-specific questions where fresh mentions are the evidence. So do this:

• Keep a real, active X account for your brand. Post about your category and your products, not just promotions.
• Earn mentions from real accounts in your space. Replies, quotes, and genuine discussion carry more weight than your own posts.
• Correct mistakes about your brand on X publicly. Grok reads those corrections, and wrong facts travel fast on the platform.
• Keep your brand facts consistent between your site and your X profile: one canonical brand name, one location, one description. You cannot fix old press, but you can fix everything you own.
• Do not buy fake engagement. Grok's whole value proposition is reading X directly, so gaming it is gaming the model's raw input, and it reads poorly.

Then set your expectations correctly: X presence helps Grok answer questions about what is happening now and what people think. It does not make Grok recommend your product to a buyer comparing options. That is decided by your site, your schema, and your reviews.

Step 4: Verify honestly, knowing verification is weak

There is no official verification path for Grok. No webmaster tool, no IP list, no documented token. The only method anyone has is a honeypot canary URL: a page linked from nowhere, pasted into Grok chat, watched in your server logs.

Say the limits plainly. A fetch of the canary page proves someone fetched it, not that Grok did, because the fetches arrive as ordinary browser user agents from proxy IPs. The absence of a hit proves nothing at all. This is the best anyone can do with Grok, because xAI has published nothing to check against. Any vendor promising confirmed Grok inclusion is selling something they cannot verify.

What this guide does not cover

This gets you readable to Grok. It does not keep you readable. Grok's fetch behavior shifts without any announcement, your catalog drifts, and xAI publishes nothing you can monitor against, so the monthly re-check is the real work. And getting recommended is not getting bought from: letting the agent complete the purchase is a separate job.

For the ChatGPT version of this playbook, see our guide to getting ChatGPT to recommend your brand. For Claude's, see our guide to getting Claude to recommend your brand.