# llms.txt checker

Check that your /llms.txt is reachable, served as text and follows the llmstxt.org layout, and see whether the pages it points to serve clean Markdown that language models can read.

Run it at https://crawlwise.dev/tools/llms-txt-checker: paste a public URL, no account needed. The scan runs the full CrawlWise audit and returns a private report.

## What this checker tests

- llms.txt directory is present (`agent.llms-txt`)
- Markdown is available for agents (`agent.markdown`)
- Markdown is advertised and negotiated correctly (`agent.markdown-negotiation`)

## Why check llms.txt and Markdown together

`llms.txt` is a directory: a short Markdown file that tells a language model what your site is and
links to its most useful pages. The links work best when they point to Markdown versions of those
pages, so CrawlWise checks the file and the Markdown it leads to in the same scan.

The report shows:

- whether `/llms.txt` answers with `200` and a text content type, not a single-page app's HTML
  shell that answers every path;
- whether it starts with an H1 and lists at least three Markdown links, and whether an
  `llms-full.txt` sits next to it;
- whether the scanned page has a Markdown version, found through a `.md` URL, a
  `<link rel="alternate" type="text/markdown">` or an `Accept: text/markdown` request;
- whether the Markdown names the HTML page as canonical and keeps the page's text.

## Common mistakes

- **An HTML page at /llms.txt.** Many frameworks answer every unknown path with the app shell.
  The file looks present in a browser but is not Markdown.
- **No H1.** It is the only required part of the file.
- **Links to HTML pages full of navigation.** Point to Markdown where you have it.
- **Using llms.txt to block bots.** It grants or denies nothing; use robots.txt for access.

## Publish one in ten minutes

Write a Markdown file with your site name as the H1, a one-sentence summary in a blockquote and a
list of your key pages, and serve it at `/llms.txt`. The [llms.txt guide](/guides/llms-txt) has
an example, and [Markdown for agents](/guides/markdown-for-agents) shows how to serve `.md`
versions with the right headers. CrawlWise publishes its own at
[/llms.txt](https://crawlwise.dev/llms.txt).

## FAQ

### What does an llms.txt validator check?

The proposal at llmstxt.org requires one thing, an H1 with the site name. It recommends a blockquote summary, then H2 sections of Markdown links with optional notes. CrawlWise checks that the file exists at the root, is not an HTML page, starts with an H1 and lists at least three Markdown links. It does not fetch the links.

### Do ChatGPT or Google read llms.txt?

No major AI search engine has said it reads llms.txt. Some coding assistants and agent tools do. Treat it as a cheap, optional directory for agents, not as a ranking factor.

### Is a missing llms.txt a problem?

Rarely. The check is experimental and worth little in the CrawlWise score. Crawler access and content that does not need JavaScript matter far more for AI search.

### What is llms-full.txt?

A convention, not part of the proposal: the Markdown of every page in one file, so an agent can load a whole site in one request. It is useful for documentation sites.

## Related guides

- [llms.txt: what it is and whether you need one](https://crawlwise.dev/guides/llms-txt.md)
- [Serve Markdown to AI agents: .md pages and headers](https://crawlwise.dev/guides/markdown-for-agents.md)

---

Last updated 14 November 2026.

HTML version: <https://crawlwise.dev/tools/llms-txt-checker>

Every CrawlWise page in Markdown: <https://crawlwise.dev/llms.txt>
