llms.txt: what it is and whether you need one

llms.txt is a short Markdown file at the root of a site that tells a language model what the site is and where its most useful pages are. It is cheap to publish and useful for some sites, but it is a proposal, not a standard, and no major AI search engine has said it reads one.

This guide explains what the file contains, what it does and does not do, and how to write one.

Where it comes from

Jeremy Howard proposed /llms.txt on September 3, 2024. The proposal, now in its second version, lives at llmstxt.org. The idea is practical: a model working within a limited context window does better with a concise, link-rich summary than with a site’s full HTML, navigation and scripts included.

It is not robots.txt. llms.txt does not grant or deny access to anything; it is a directory. Use robots.txt and your firewall for access, as described in AI crawlers in robots.txt.

What goes in the file

The proposal defines the sections in this order:

1. An H1 with the name of the site or project. This is the only required section.
2. A blockquote with a short summary.
3. Any number of paragraphs or lists, but no headings, with details a model should know.
4. H2 sections that are lists of links. Each item is a Markdown link, optionally followed by a colon and a note.
5. An Optional H2 section for secondary links that a model can skip when context is short.

A small example:

# Acme Analytics

> Acme Analytics is a privacy-friendly web analytics service for small businesses.

Acme does not use cookies. Pricing is per site, billed monthly.

## Docs

- [Quick start](https://acme.example/docs/quick-start.md): add the script and see your first visit
- [API reference](https://acme.example/docs/api.md): read stats over HTTPS with an API key
- [Pricing](https://acme.example/pricing.md): plans, limits and what counts as a page view

## Optional

- [Changelog](https://acme.example/changelog.md): releases since 2023

The proposal also suggests a Markdown version of each linked page, at the same URL with .md added, or with the extension replaced by .md. A link list that points to clean Markdown is far more useful to a model than one that points to HTML. See Serve Markdown to AI agents for how to publish those.

What about llms-full.txt?

llms-full.txt is not part of the proposal. It is a convention, generated by documentation platforms such as Mintlify, for a single file that contains the full text of a site’s documentation. It is worth adding for documentation that someone may want to load into a model in one go. For a marketing site it adds little.

Who reads it

Be clear-eyed about this part.

• No AI search engine documents reading it. OpenAI’s, Anthropic’s and Perplexity’s crawler documentation does not mention llms.txt.
• Google says you do not need it. Its guidance for AI Overviews and AI Mode says: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.”
• AI companies publish their own. OpenAI and Anthropic both serve llms.txt files for their developer documentation, linking to Markdown pages.
• Tools and agents use it. The clearest value is when a person or a coding agent is pointed at your site and needs to find the right page fast, which is why documentation sites adopted it first.
• Lighthouse checks for it. Chrome’s Lighthouse has an audit that flags a server error when fetching llms.txt and marks a missing file as not applicable, because the file is optional.

In short: publish it if you have documentation, an API or a product that people ask assistants about. Do not expect it to change rankings or citations on its own.

Mistakes to avoid

• Serving HTML. A missing route that falls through to your HTML 404 page, or to the home page with status 200, is worse than no file. Serve text/plain or text/markdown.
• Linking to the wrong format. Link Markdown versions where you have them, and use absolute URLs.
• Listing everything. The file is a curated index. A hundred links with no notes is just a worse sitemap.
• Letting it go stale. Generate it from the same source as your sitemap or navigation, so a new page does not need a manual edit.
• Treating it as access control. Keeping a URL out of llms.txt hides nothing.

Publish one

Write the file by hand for a small site, or generate it from your pages:

• Static sites and Rails: a file in public/llms.txt, or a route that renders it from your page list.
• Next.js: a Route Handler at app/llms.txt/route.ts that returns the text with Content-Type: text/plain; charset=utf-8.
• Documentation platforms: many generate it already; check before writing your own.

Then check it from outside:

curl -sS -D - https://example.com/llms.txt -o /dev/null | grep -i '^content-type'
curl -sS https://example.com/llms.txt | head -20

A CrawlWise scan fetches /llms.txt, checks that it is not HTML, starts with an H1 and links to at least three pages, and notes whether llms-full.txt exists. Because the format is experimental, the check carries a small weight in the score.

Sources

• Jeremy Howard: The /llms.txt file
• Google: AI features and your website
• Chrome: Lighthouse llms.txt audit
• OpenAI: OpenAI Developers llms.txt
• Anthropic: Claude developer documentation llms.txt
• Mintlify: llms.txt