In Salesforce Commerce Cloud, robots.txt is developer-managed and instance-specific. In Business Manager go to Merchant Tools, then SEO & Discoverability, then Robots. Select the instance type, choose the Custom robots.txt Definition option, and write the complete file content there.
Why Salesforce Commerce Cloud is different
There are two ways to serve the file, and they do not mix. You can type a custom definition into Business Manager (Salesforce's recommended route), or you can place a robots.txt file in the cartridge/static/default directory of your storefront cartridge. There can only be one robots.txt per cartridge, so pick one mechanism and stay with it.
The definition is per instance type. Staging and production want different rules: staging should block everything, production should welcome the retrieval crawlers. Salesforce's own docs recommend authoring the file on a staging instance and replicating the site preferences to production.
The hostname alias must be configured, or the file is not served under your domain at all. A correct definition with a missing alias produces a 404 at /robots.txt.
Storefront Password Protection auto-generates a block-all robots.txt that overrides your definition. If password protection is on, your custom file never takes effect, no matter how carefully you wrote it.
Salesforce's own documentation names GPTBot, ClaudeBot, and PerplexityBot as crawlers the platform serves the file to. The AI angle here is not theoretical: the platform expects these bots to request the file.
The steps, in order
- Check what is served today, on the right instance. Open https://[yourdomain.com]/robots.txt in a browser, replacing yourdomain.com with your actual domain, on staging and on production separately. The two should look different.
- Choose your mechanism. For most teams, the Custom robots.txt Definition in Business Manager is the right call. Use the deployed-cartridge file only if you need the same file shared across multiple sites.
- Write the definition. In Business Manager go to Merchant Tools, then SEO & Discoverability, then Robots. Select the instance type. For staging, a full block is the correct content:
User-agent: *
Disallow: /For production, write the full file with your defaults plus the AI retrieval crawler groups near the top, before any broad disallow rules:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: meta-webindexer
Allow: /
User-agent: bingbot
Allow: /
User-agent: *
Disallow: /account*
Disallow: /cart*
Disallow: /checkout*- Replicate staging site preferences to production. Authoring on production directly skips the staging check and risks pushing staging's block-all rules live.
- Verify on the production hostname. If the old file persists, invalidate the static content cache and reload /robots.txt.
The fundamentals
Three things work the same on every platform:
- •What goes in an llms.txt file and where to publish it on Salesforce Commerce Cloud, see our llms.txt guide
- •The product markup AI tools read on the page, see our product schema guide
- •The full crawler token reference and what each allow or block costs, see our robots.txt guide
A hypothetical example
Say you run a fashion brand on Salesforce Commerce Cloud. You write the production definition in Business Manager, verify it on your staging hostname, and call it done. A scan of the production domain still shows a block-all file. The definition was fine; the site preferences were never replicated from staging to production, and the hostname alias check was never run. On this platform the file is never "just" a file. Instance, replication, and alias are part of the edit.
What this guide does not cover
The file is only the declaration. If a bot manager or the Commerce Cloud edge blocks an AI crawler before it reads robots.txt, the file's rules do not matter. That edge check, plus ongoing monitoring for drift and the transaction layer that lets an agent complete a purchase, are separate work.
Frequently Asked Questions
Where are robots.txt files managed in Business Manager?
Can I have different crawler rules for staging and production?
Why are bots still blocked after updating robots.txt in SFCC?
Check your own brand now
Run the free scan at aizii.ai/aeo-scan. One click, no signup. It shows which AI crawlers can reach your site today and which ones are blocked.
Run the free scan