The two files sit side by side at the root of a site and are easy to confuse, but they answer different questions. robots.txt answers “may you come in?” It is a list of rules per crawler, written as User-agent, Allow and Disallow lines, and it has been an IETF standard since 2022 (RFC 9309). Crawlers that follow it read it before they fetch anything else.
llms.txt answers “what is here, and what should you read first?” It is a Markdown file proposed at llmstxt.org: a title, a one-line summary and lists of links with a short description each. No search engine or AI company has said that its systems read it, and we do not claim any do. It is cheap to publish and low-risk, which is why we publish it.
Because they do different jobs, one never replaces the other. A site that blocks an AI crawler in robots.txt is not unblocked by a good llms.txt, and a site with no llms.txt loses nothing in crawling. The common mistake runs the other way: a robots.txt rule written years ago, for a different crawler, that now blocks the AI systems a company wants to be cited by.
robots.txt also does not remove a page from search. A URL that is disallowed can still be indexed from links elsewhere, without its content, because the crawler never read the page. To keep a page out of the index, the page has to be crawlable and carry noindex.