Aizii Learn · AI Search

[← Back to Learn](https://aizii.ai/learn)

# How to edit robots.txt on BigCommerce for AI crawlers

BigCommerce puts a plain-text robots.txt editor in the store control panel. Go to Settings, then Website, scroll down to the Search Engine Robots section, and add your AI crawler groups below the defaults. Save.

## Why BigCommerce is different

No template language, no plugin, no code. What you type in the text box is what gets served. That makes it the simplest edit of any hosted platform, and it means syntax errors are entirely yours to make.

BigCommerce's default file is long and specific. It disallows account, cart, checkout, login, order-status, search, and wishlist paths, and it declares your sitemap at the bottom. Keep every default line. Your job is to append AI crawler rules, not to rebuild the file.

The default file contains two separate User-agent: \* groups with a Sitemap line between them. Put your AI crawler groups before the first broad group so the specific rules are read first, and leave the Sitemap line where it is.

## The steps, in order

1.  Check what is served today. Open https://\[yourdomain.com\]/robots.txt in a browser, replacing yourdomain.com with your actual domain. Read the full default file so you know what must stay.

1.  Open the editor. In the BigCommerce control panel go to Settings, then Website, and scroll to the Search Engine Robots section.

1.  Add the AI retrieval crawler groups above the first User-agent: \* group:

```
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: meta-webindexer
Allow: /

User-agent: bingbot
Allow: /
```

1.  Save and verify. Click Save, then reload https://\[yourdomain.com\]/robots.txt and confirm your groups appear at the top with all the defaults intact below them.

One note for multi-storefront setups: BigCommerce also exposes this setting through its API (PUT on the storefront robotstxt settings endpoint), which supports per-channel values. If you run more than one storefront, that is the route to different rules per storefront.

### The fundamentals

Three things work the same on every platform:

-   •What goes in an llms.txt file and where to publish it on BigCommerce, see our [llms.txt guide](https://aizii.ai/learn/ai-search/how-to-write-a-good-llms-txt-file)
-   •The product markup AI tools read on the page, see our [product schema guide](https://aizii.ai/learn/ai-search/how-to-add-product-schema-markup-for-ai-search)
-   •The full crawler token reference and what each allow or block costs, see our [robots.txt guide](https://aizii.ai/learn/ai-search/how-to-write-robots-txt-for-ai-search)

### A hypothetical example

Say you run an outdoor-gear store on BigCommerce. Wanting a "clean" file, you replace the whole editor contents with just the AI allowlist and save. Two weeks later your filtered category URLs are getting crawled aggressively and your cart pages start showing up in places they should not. The default disallows you wiped out were doing real work. The fix is to restore BigCommerce's default lines and append the AI groups instead. On BigCommerce, the defaults are the file; your rules are the addition.

## What this guide does not cover

The file is only the declaration. If a bot manager or CDN blocks an AI crawler at the edge before it reads robots.txt, the file's rules do not matter. That edge check, plus ongoing monitoring for drift and the transaction layer that lets an agent complete a purchase, are separate work.

## Frequently Asked Questions

Where is the robots.txt editor located in BigCommerce?

Log into your control panel, go to Settings, choose Website, and scroll down to the Search Engine Robots field.

Does BigCommerce provide default rules for AI crawlers?

No. BigCommerce includes standard search rules, so you must explicitly add sections for AI bots like PerplexityBot and OAI-SearchBot.

How fast do robots.txt changes take effect on BigCommerce?

Changes save immediately on the server. AI crawlers will see the updated directives on their subsequent fetch.

## Check your own brand now

Run the free scan at aizii.ai/aeo-scan. One click, no signup. It shows which AI crawlers can reach your site today and which ones are blocked.

[Run the free scan](https://aizii.ai/aeo-audit)