"Custom website" describes how the site was built, not where it runs, so the editing method follows your hosting. Identify the hosting layer first: a static host, a platform-as-a-service, or a server you manage. The rule is the same in all three: find the file that is actually served at /robots.txt and edit that one.
Why a custom website is different
There is no single path, because there is no single platform. The method is determined by the hosting layer, and hand-built sites hide an extra step: confirming which file the server actually serves. A robots.txt in your repo means nothing if your router or build pipeline never deploys it.
On static hosts (Netlify, Vercel, Cloudflare Pages), robots.txt is a static file in the repo's publish directory, the same directory index.html is served from. It deploys with the site. Change the file, redeploy, and the new rules go live. No redeploy, no change.
Single-page apps are the classic trap. A catch-all route that rewrites every request to index.html can swallow /robots.txt requests, handing crawlers your app shell instead of rules. The file must be served before the catch-all in your routing order.
On a platform-as-a-service (Render, Fly.io, Heroku), there is no shared static directory. Serve robots.txt from your app code or your static middleware, and treat it like any other route you own.
On a VPS or dedicated box that you manage yourself, this is a self-hosted site. The file lives in the document root. See our guide to editing robots.txt on a self-hosted site.
The steps, in order
- Check what is served today. Open https://[yourdomain.com]/robots.txt in a browser, replacing yourdomain.com with your actual domain. If you get your app shell, a 404, or an HTML page, the request is not reaching a robots.txt file at all.
- Identify your hosting layer. Static host, platform-as-a-service, or your own server. The steps below branch from here.
- Static host: add or edit robots.txt in the publish directory of your repo (where index.html is served from). Commit, redeploy, and reload the URL to confirm.
- Single-page app: check your routing order. Make sure /robots.txt is served as a static file before the catch-all rewrite to index.html. A rewrite rule that fires first will shadow the file no matter how correct it is.
- Platform-as-a-service: add a route or static middleware in your app that serves the file at /robots.txt with a text/plain content type. Deploy and verify.
- Your own server: edit the file in the document root. See our guide to editing robots.txt on a self-hosted site for the full method.
- Add the AI retrieval crawler groups near the top, before any broad disallow rules:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: meta-webindexer
Allow: /
User-agent: bingbot
Allow: /- Verify from the outside. Reload https://[yourdomain.com]/robots.txt and confirm the groups are present with the right content type. What your repo contains and what the URL serves are two different facts.
The fundamentals
Three things work the same on every platform:
- •What goes in an llms.txt file and where to publish it on a custom website, see our llms.txt guide
- •The product markup AI tools read on the page, see our product schema guide
- •The full crawler token reference and what each allow or block costs, see our robots.txt guide
A hypothetical example
Say you run a SaaS marketing site built with a static generator and deployed on a static host. Your developer adds robots.txt to the repo root, where the source files live, and pushes. The URL still 404s. The publish directory is a build output folder, and the file never made it into the build. Moving it next to the built index.html and redeploying fixed it in one deploy. On a custom site, the question is never "did I write the file" but "did the file reach the URL".
What this guide does not cover
The file is only the declaration. If a bot manager or CDN blocks an AI crawler at the edge before it reads robots.txt, the file's rules do not matter. That edge check, plus ongoing monitoring for drift and the transaction layer that lets an agent complete a purchase, are separate work.
Frequently Asked Questions
How do I manage robots.txt in modern frameworks like Next.js?
Can I return different robots.txt rules based on environment?
Does Vercel or Netlify cache robots.txt?
Check your own brand now
Run the free scan at aizii.ai/aeo-scan. One click, no signup. It shows which AI crawlers can reach your site today and which ones are blocked.
Run the free scan