WooCommerce runs on WordPress, and WordPress generates robots.txt on the fly. There is no physical file by default. You edit it through your SEO plugin's file editor, or you place a physical robots.txt in the site root, which overrides the virtual one completely.
Why WooCommerce is different
Two files can exist and only one wins. A physical robots.txt sitting in your site's root directory (usually public_html) overrides the virtual one entirely. Editing the plugin editor while a forgotten physical file sits in the root is the classic reason changes appear to do nothing.
WordPress's virtual defaults only block wp-admin and wp-includes. WooCommerce adds no crawler rules of its own. Cart, checkout, and account pages stay out of indexes through other means, not through this file. That makes your AI crawler groups purely additive: you are adding allow rules to a file that never blocked them.
Plugin editors are not interchangeable. Yoast SEO and Rank Math each keep their own editor, and each writes the file its own way. Know which plugin is authoritative on your site before you edit.
The steps, in order
- Check what is served today. Open https://[yourdomain.com]/robots.txt in a browser, replacing yourdomain.com with your actual domain. Note whether the rules look like WordPress defaults or something custom.
- Check for a physical file first. In your hosting file manager or over FTP, look in the site root (public_html) for a file named robots.txt. If one exists, that file wins over everything else. Edit that file directly, or delete it if you want the virtual file (and your plugin editor) to take over.
- If there is no physical file, use your SEO plugin's editor:
- With Yoast SEO: in the WordPress dashboard go to SEO, then Tools, then File editor. If no robots.txt exists, Yoast offers to create one. Edit the rules and click Save changes to robots.txt.
- With Rank Math: go to Rank Math, then General Settings, then Edit robots.txt. Make your changes and save.
- Add the AI retrieval crawler groups near the top, before any broad disallow rules:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: meta-webindexer
Allow: /
User-agent: bingbot
Allow: /Keep WordPress's default blocks on wp-admin and wp-includes. Do not disallow wp-content wholesale: crawlers need the CSS and JavaScript inside it to render your product pages.
- Save and verify. Reload https://[yourdomain.com]/robots.txt and confirm your groups are present.
The fundamentals
Three things work the same on every platform:
- •What goes in an llms.txt file and where to publish it on WooCommerce, see our llms.txt guide
- •The product markup AI tools read on the page, see our product schema guide
- •The full crawler token reference and what each allow or block costs, see our robots.txt guide
A hypothetical example
Say you run a bike-parts store on WooCommerce with Yoast installed. You add the AI crawler allowlist through Yoast's file editor, save, and check the URL. Nothing changed. The file still shows old rules. The reason: a physical robots.txt left over from a site migration was sitting in public_html, and it overrode everything Yoast wrote. Deleting the stale physical file let the plugin's version take over, and the new rules appeared immediately. On WooCommerce, always check the root directory before assuming the editor is broken.
What this guide does not cover
The file is only the declaration. If a security plugin or the hosting firewall blocks an AI crawler at the edge before it reads robots.txt, the file's rules do not matter. That edge check, plus ongoing monitoring for drift and the transaction layer that lets an agent complete a purchase, are separate work.
Frequently Asked Questions
Does WordPress have a physical robots.txt file by default?
What happens if a physical robots.txt exists in the root folder?
Can SEO plugins manage robots.txt for AI bots on WooCommerce?
Check your own brand now
Run the free scan at aizii.ai/aeo-scan. One click, no signup. It shows which AI crawlers can reach your site today and which ones are blocked.
Run the free scan