Comparisons

llms.txt vs robots.txt

robots.txt tells crawlers which URLs on a site they may fetch, and the main search and AI crawlers honour it. llms.txt is an optional, human-readable map that tells language models what the site is and which pages matter; it grants or blocks nothing.

The two files sit side by side at the root of a site and are easy to confuse, but they answer different questions. robots.txt answers “may you come in?” It is a list of rules per crawler, written as User-agent, Allow and Disallow lines, and it has been an IETF standard since 2022 (RFC 9309). Crawlers that follow it read it before they fetch anything else.

llms.txt answers “what is here, and what should you read first?” It is a Markdown file proposed at llmstxt.org: a title, a one-line summary and lists of links with a short description each. No search engine or AI company has said that its systems read it, and we do not claim any do. It is cheap to publish and low-risk, which is why we publish it.

Because they do different jobs, one never replaces the other. A site that blocks an AI crawler in robots.txt is not unblocked by a good llms.txt, and a site with no llms.txt loses nothing in crawling. The common mistake runs the other way: a robots.txt rule written years ago, for a different crawler, that now blocks the AI systems a company wants to be cited by.

robots.txt also does not remove a page from search. A URL that is disallowed can still be indexed from links elsewhere, without its content, because the crawler never read the page. To keep a page out of the index, the page has to be crawlable and carry noindex.

Side by side

Aspectrobots.txtllms.txt
What it doesAllows or disallows crawling, per user agentDescribes the site and lists the pages that matter
StatusIETF standard, RFC 9309 (2022)A proposal published at llmstxt.org
FormatPlain-text rules: User-agent, Allow, Disallow, SitemapMarkdown: title, summary, sections of links
Who reads itSearch crawlers and the AI crawlers that publish their user agentsNo engine has confirmed that it does
If it is missingEvery URL may be crawledNothing changes in crawling or indexing
If it is wrongA stray Disallow can keep crawlers off the whole siteAn outdated map describes pages that no longer exist
Can it keep a page out of search?No: use noindex on a crawlable pageNo

When to choose robots.txt

  • You need to keep crawlers off URLs that waste their time, such as internal search results or endless filter combinations.
  • You want to allow or refuse a named AI crawler, such as GPTBot or ClaudeBot, separately from search crawlers.
  • You want to point crawlers at your sitemap with a Sitemap line.

When to choose llms.txt

  • Your site is large or its navigation is heavy, and a short plain-text summary would help a model find the pages that matter.
  • You can generate the file from the same route list as your sitemap, so it cannot drift out of date.
  • You have already checked robots.txt, because llms.txt does nothing for a crawler that is blocked.

An example from our own sites

On audacian.eu, robots.txt allows every crawler and names the ones we welcome explicitly: Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot and Google-Extended. A comment at the top points at the sitemap, llms.txt and llms-full.txt. The two llms files are generated at build time from the same route list as the sitemap, so a page we remove disappears from all three at once, and a page marked noindex is left out of llms-full.txt. Our free AI search readiness checker reads both files on any site you enter: it reports which AI crawlers robots.txt blocks and whether llms.txt exists.

Questions

Does llms.txt replace robots.txt?
No. robots.txt decides what crawlers may fetch; llms.txt only describes the site. A crawler blocked in robots.txt is still blocked, whatever llms.txt says.
Can robots.txt keep a page out of Google?
Not reliably. A disallowed URL can still be indexed from links, without its content. To keep a page out of the index, let it be crawled and give it a noindex tag or header.
Do ChatGPT or Perplexity read llms.txt?
Neither company has said so, and we make no claim that they do. What they do publish is the user agents of their crawlers, which you can allow or block in robots.txt.

Tell us what you're launching.

Book a 20-minute intro call, or email a few lines about what you're launching. We'll reply with a rough plan and a price range, whether or not we end up working together.