Aizii Learn · AI Search

[← Back to Learn](https://aizii.ai/learn)

# llms.txt vs robots.txt: what's the difference

robots.txt is access control. It tells crawlers which parts of your site they may fetch: it grants or denies permission, and it does nothing more. llms.txt is a content file: a markdown summary you write for the models, like a concise briefing document about your site. It cannot block or allow anything. Different jobs, both live at the site root, and they are complementary, not competing.

## Why they get confused

1.  Both live at the root, both end in .txt, and both appear on every AI search checklist. It is easy to file them as one job.
2.  The name llms.txt sounds like a control file. It is not: nothing reads it to decide permission. It is the opposite of a gate; it is an invitation to read.
3.  robots.txt looks like it says something about your site. It does not: it names crawlers and paths. A perfect robots.txt tells a model nothing about what you sell.

## Step 1: Get robots.txt right first

Check \[yourdomain.com\]/robots.txt, replacing yourdomain.com with your actual domain. Confirm it exists, and confirm the AI crawler groups are allowed. Create or fix it before anything else: a blocked declared crawler never reaches your pages, and it never reaches your llms.txt either.

## Step 2: Add llms.txt at the root

Check whether \[yourdomain.com\]/llms.txt loads. If it does not, create it: a short markdown file in your own words with your name, what you sell, who it is for, and how to buy. The full format and a complete sample live in the llms.txt guide; the shape is simple:

```
# [Your Shop Name]

[One line on what you sell and who it is for.]

## Products

- [Product line one]: [one-line description]
- [Product line two]: [one-line description]

## How to buy

[Where and how a customer completes a purchase.]
```

## Step 3: Keep both current

New product line, new page, new policy: update the site and update the llms.txt summary. robots.txt only changes when your crawler policy changes. The briefing goes stale faster than the gate, so check it more often.

### The fundamentals

These mechanics are shared across every engine and covered in full in their own guides:

-   •The full llms.txt format, a complete sample, and per-platform publish steps, see our [guide to writing a good llms.txt file](https://aizii.ai/learn/ai-search/how-to-write-a-good-llms-txt-file)
-   •The complete crawler rule reference, see our [robots.txt guide for AI search](https://aizii.ai/learn/ai-search/how-to-write-robots-txt-for-ai-search)
-   •The allow rules in one paste-ready block, see our guide to allowing AI crawlers in robots.txt

### Say you run a bike shop in Austin

Your robots.txt allows the AI crawlers. Your llms.txt says, in plain markdown: "We are a bike shop in Austin. We sell gravel bikes and e-bikes, and our service department does repairs and fittings." A crawler fetching your site gets permission from one file and a briefing from the other. Block the crawlers in robots.txt and the declared bots never reach the briefing. Delete the llms.txt and the crawler reads your pages raw, with no summary. Each file does its own job.

## What this guide does not cover

This guide draws the boundary between the two files. The full llms.txt format lives in its own guide, the full crawler reference lives in the robots.txt guide, and monitoring either file for drift is a separate job.

## Frequently Asked Questions

Can an AI bot read llms.txt if robots.txt disallows it?

No. Respectful crawlers check robots.txt before fetching any URL on your domain, including llms.txt. If your site disallows crawling, the bot will never read llms.txt.

Which file should I set up first?

Set up robots.txt first to ensure AI crawlers have permission to reach your pages. Then deploy llms.txt to help them understand your brand structure.

Is llms.txt an official web standard?

It is an emerging community convention widely adopted by AI developers, but not yet an official W3C or IETF standard.

## Check your own site now

Run the free scan. One click, no signup, and it shows whether AI crawlers can reach your site and what they find: aizii.ai/aeo-scan

[Run the free scan](https://aizii.ai/aeo-audit)