llms.txt

llms.txt is a proposed plain-text file placed at the root of a website that describes the site's content and structure to large language models and AI agents, similar in spirit to robots.txt but oriented toward reading and understanding rather than crawl control. It is one of the checks in the Lighthouse agentic browsing category.

llms.txt is a text file, served at the root of a domain (for example, https://example.com/llms.txt), that gives large language models and AI agents a curated, machine-readable summary of the site. The format is Markdown-based: a top-level H1 with the site name, an optional short description, and one or more H2 sections that link to the most important pages (documentation, product pages, policies, glossaries).

Unlike robots.txt, llms.txt is not about crawl permission. It is about making the most useful content easy for a language model to find and cite, in a form that does not require the model to parse a full HTML page tree. The proposal was published by Jeremy Howard in late 2024 and has been adopted by a growing number of documentation sites and AI-facing publishers through 2025 and 2026.