# llms.txt: what it is and whether you need one `llms.txt` is a short Markdown file at the root of a site that tells a language model what the site is and where its most useful pages are. It is cheap to publish and useful for some sites, but it is a proposal, not a standard, and no major AI search engine has said it reads one. This guide explains what the file contains, what it does and does not do, and how to write one. ## Where it comes from Jeremy Howard proposed `/llms.txt` on September 3, 2024. The proposal, now in its second version, lives at [llmstxt.org](https://llmstxt.org/). The idea is practical: a model working within a limited context window does better with a concise, link-rich summary than with a site's full HTML, navigation and scripts included. It is not robots.txt. `llms.txt` does not grant or deny access to anything; it is a directory. Use robots.txt and your firewall for access, as described in [AI crawlers in robots.txt](/guides/ai-crawlers-robots-txt). ## What goes in the file The proposal defines the sections in this order: 1. **An H1 with the name of the site or project.** This is the only required section. 2. **A blockquote** with a short summary. 3. **Any number of paragraphs or lists**, but no headings, with details a model should know. 4. **H2 sections that are lists of links.** Each item is a Markdown link, optionally followed by a colon and a note. 5. **An `Optional` H2 section** for secondary links that a model can skip when context is short. A small example: ```markdown # Acme Analytics > Acme Analytics is a privacy-friendly web analytics service for small businesses. Acme does not use cookies. Pricing is per site, billed monthly. ## Docs - [Quick start](https://acme.example/docs/quick-start.md): add the script and see your first visit - [API reference](https://acme.example/docs/api.md): read stats over HTTPS with an API key - [Pricing](https://acme.example/pricing.md): plans, limits and what counts as a page view ## Optional - [Changelog](https://acme.example/changelog.md): releases since 2023 ``` The proposal also suggests a Markdown version of each linked page, at the same URL with `.md` added, or with the extension replaced by `.md`. A link list that points to clean Markdown is far more useful to a model than one that points to HTML. See [Serve Markdown to AI agents](/guides/markdown-for-agents) for how to publish those. ### What about llms-full.txt? `llms-full.txt` is not part of the proposal. It is a convention, generated by documentation platforms such as Mintlify, for a single file that contains the full text of a site's documentation. It is worth adding for documentation that someone may want to load into a model in one go. For a marketing site it adds little. ## Who reads it Be clear-eyed about this part. - **No AI search engine documents reading it.** OpenAI's, Anthropic's and Perplexity's crawler documentation does not mention `llms.txt`. - **Google says you do not need it.** Its guidance for AI Overviews and AI Mode says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." - **AI companies publish their own.** OpenAI and Anthropic both serve `llms.txt` files for their developer documentation, linking to Markdown pages. - **Tools and agents use it.** The clearest value is when a person or a coding agent is pointed at your site and needs to find the right page fast, which is why documentation sites adopted it first. - **Lighthouse checks for it.** Chrome's Lighthouse has an audit that flags a server error when fetching `llms.txt` and marks a missing file as not applicable, because the file is optional. In short: publish it if you have documentation, an API or a product that people ask assistants about. Do not expect it to change rankings or citations on its own. ## Mistakes to avoid - **Serving HTML.** A missing route that falls through to your HTML 404 page, or to the home page with status 200, is worse than no file. Serve `text/plain` or `text/markdown`. - **Linking to the wrong format.** Link Markdown versions where you have them, and use absolute URLs. - **Listing everything.** The file is a curated index. A hundred links with no notes is just a worse sitemap. - **Letting it go stale.** Generate it from the same source as your sitemap or navigation, so a new page does not need a manual edit. - **Treating it as access control.** Keeping a URL out of `llms.txt` hides nothing. ## Publish one Write the file by hand for a small site, or generate it from your pages: - **Static sites and Rails:** a file in `public/llms.txt`, or a route that renders it from your page list. - **Next.js:** a Route Handler at `app/llms.txt/route.ts` that returns the text with `Content-Type: text/plain; charset=utf-8`. - **Documentation platforms:** many generate it already; check before writing your own. Then check it from outside: ```sh curl -sS -D - https://example.com/llms.txt -o /dev/null | grep -i '^content-type' curl -sS https://example.com/llms.txt | head -20 ``` A [CrawlWise scan](/) fetches `/llms.txt`, checks that it is not HTML, starts with an H1 and links to at least three pages, and notes whether `llms-full.txt` exists. Because the format is experimental, the check carries a small weight in the score. ## Sources - Jeremy Howard: [The /llms.txt file](https://llmstxt.org/) - Google: [AI features and your website](https://developers.google.com/search/docs/appearance/ai-features) - Chrome: [Lighthouse llms.txt audit](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt) - OpenAI: [OpenAI Developers llms.txt](https://developers.openai.com/llms.txt) - Anthropic: [Claude developer documentation llms.txt](https://platform.claude.com/docs/llms.txt) - Mintlify: [llms.txt](https://www.mintlify.com/docs/ai/llmstxt) --- Last updated 3 November 2026. HTML version: Every CrawlWise page in Markdown: