You own the server, so you own the file. Create or edit robots.txt in the web server's document root as plain text, and it is served at https://[yourdomain.com]/robots.txt, replacing yourdomain.com with your actual domain, on the next request.
Why self-hosted is different
No panel, no template, no plugin. The file is the configuration. That removes every platform quirk and replaces it with one responsibility: the file on disk is exactly what crawlers read.
You can verify end to end in a way hosted platforms never allow. Fetch the file with curl, request a page with a bot user-agent to confirm the edge lets it through, grep your access logs for the crawler tokens, and validate the hits against the vendors' published IP ranges. The pillar guide has the exact log command and the validation sources.
The server config can shadow the file. An nginx location block or an Apache rewrite that handles /robots.txt, or a CDN caching an old copy, means your edits do not take effect until you fix the config or purge the cache. The file being correct and the file being served are two different facts. Check both.
One file per server. Behind a load balancer, keep it identical on every node, or better, serve it from your deploy pipeline so it cannot drift.
The steps, in order
- Check what is served today. In a terminal, run:
curl https://[yourdomain.com]/robots.txtreplacing yourdomain.com with your actual domain. Read the current rules.
- Edit the file in the document root. Open robots.txt in the web server's document root (for example, /var/www/html/robots.txt on a typical nginx or Apache setup) in a plain-text editor. If no file exists, create one. Save as plain text with UTF-8 encoding.
- Put the AI retrieval crawler groups near the top, before any broad disallow rules:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: meta-webindexer
Allow: /
User-agent: bingbot
Allow: /- Verify the serve, not just the file. Fetch the URL again with curl and confirm the new groups appear. Check the content type is text/plain. If you run a CDN, purge the cache for /robots.txt or wait out the cache TTL.
- Confirm crawlers actually arrive. After a week, grep your access logs for the tokens. No hits means something upstream is still blocking, regardless of what the file says.
The fundamentals
Three things work the same on every platform:
- •What goes in an llms.txt file and where to publish it on a self-hosted site, see our llms.txt guide
- •The product markup AI tools read on the page, see our product schema guide
- •The full crawler token reference and what each allow or block costs, see our robots.txt guide
A hypothetical example
Say you run software documentation on a self-hosted server behind a CDN. You write the file perfectly, verify it on disk, and move on. A scan a month later shows the AI crawlers still seeing the old rules. The CDN had cached /robots.txt with a long TTL and never picked up the change. One cache purge fixed it. On a self-hosted setup the file is the easy part; the cache in front of it is the part that bites.
What this guide does not cover
The file is only the declaration. If your firewall or CDN blocks an AI crawler at the edge before it reads robots.txt, the file's rules do not matter, and you are the one who configured that edge. That check, plus ongoing monitoring for drift and the transaction layer that lets an agent complete a purchase, are separate work.
Frequently Asked Questions
Where should robots.txt be located on a self-hosted server?
How do I ensure the web server serves robots.txt with the right headers?
Can I test my robots.txt syntax before deploying?
Check your own brand now
Run the free scan at aizii.ai/aeo-scan. One click, no signup. It shows which AI crawlers can reach your site today and which ones are blocked.
Run the free scan