Aizii Learn · AI Search

[← Back to Learn](https://aizii.ai/learn)

# How to edit robots.txt on Adobe Commerce for AI crawlers

Adobe Commerce generates robots.txt from the admin database, not from a file. Go to Content, then Design, then Configuration. Pick the store view, click Edit, open the Search Engine Robots tab, and type your rules into the Custom instruction of robots.txt File field. Save, then flush the cache.

## Why Adobe Commerce is different

There is no file to edit over FTP or SSH. On cloud infrastructure the content lives in the database, and a physical robots.txt in pub/media exists only as a side effect of the read-only file system. Editing files on disk does nothing to the served file.

The same screen holds a trap: the Default Robots dropdown. That dropdown controls the meta robots tag printed on your pages. It does not control the robots.txt file. Setting NOINDEX, FOLLOW in the dropdown while leaving the file untouched blocks indexing without touching a single crawler rule. These are two separate controls and they are easy to confuse.

Nothing changes until you flush the Magento cache. You can save perfect rules, reload /robots.txt, and see the old file. Save, flush, then check.

Each store view can carry its own robots.txt. That is useful for multi-brand setups and a trap for everyone else: editing the Global scope while checking a store-view URL shows you the wrong file.

On cloud infrastructure, if search-engine indexing is restricted (the usual state of staging environments), your custom content never applies. Adobe's own troubleshooting docs say the fix is on the indexing setting, not on the file.

## The steps, in order

1.  Check what is served today. Open https://\[yourdomain.com\]/robots.txt in a browser, replacing yourdomain.com with your actual domain. Note which store view you are checking if you run more than one.

1.  Open the configuration. In the admin go to Content, then Design, then Configuration. Find the store view you want to change and click Edit.

1.  Expand the Search Engine Robots tab. Leave Default Robots on INDEX, FOLLOW for a production store. That setting is the meta robots tag, not this file.

1.  Write your rules in the Custom instruction of robots.txt File field. Put the AI retrieval crawler groups near the top, before any broad disallow rules:

```
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: meta-webindexer
Allow: /

User-agent: bingbot
Allow: /
```

1.  Click Save Configuration, then flush the Magento cache. Reload https://\[yourdomain.com\]/robots.txt and confirm the new groups are present.

1.  To roll back, use the Reset to Default button in the same tab. It restores Adobe Commerce's default instructions.

### The fundamentals

Three things work the same on every platform:

-   •What goes in an llms.txt file and where to publish it on Adobe Commerce, see our [llms.txt guide](https://aizii.ai/learn/ai-search/how-to-write-a-good-llms-txt-file)
-   •The product markup AI tools read on the page, see our [product schema guide](https://aizii.ai/learn/ai-search/how-to-add-product-schema-markup-for-ai-search)
-   •The full crawler token reference and what each allow or block costs, see our [robots.txt guide](https://aizii.ai/learn/ai-search/how-to-write-robots-txt-for-ai-search)

### A hypothetical example

Say you run a specialty-foods store on Adobe Commerce. Your developer pastes the AI crawler groups into the custom-instructions field, saves, and checks the URL. The old file is still there. The rules were correct; the cache was not flushed. One cache flush later, the new groups appear. Then the team edits the Global scope and checks the US store-view URL, which carries its own older rules. On Adobe Commerce, scope and cache are part of the edit, every time.

## What this guide does not cover

The file is only the declaration. If a bot manager, Fastly VCL, or CDN rule blocks an AI crawler at the edge before it reads robots.txt, the file's rules do not matter. That edge check, plus ongoing monitoring for drift and the transaction layer that lets an agent complete a purchase, are separate work.

## Frequently Asked Questions

Where can I edit robots.txt in the Adobe Commerce admin panel?

Navigate to Content, select Configuration under Design, and expand the Search Engine Robots section.

Can I place a physical robots.txt in the pub directory instead?

Yes. Placing a physical file in pub/robots.txt takes precedence and is served directly by the web server.

Does Magento caching affect robots.txt updates?

If you edit via admin, flush the Magento full page cache and Varnish/Fastly cache to ensure crawlers receive the new content.

## Check your own brand now

Run the free scan at aizii.ai/aeo-scan. One click, no signup. It shows which AI crawlers can reach your site today and which ones are blocked.

[Run the free scan](https://aizii.ai/aeo-audit)