What this checker tests
The scan runs the full CrawlWise audit. These checks answer this page's question; the report shows each one with its evidence.
- llms.txt directory is present
agent.llms-txt - Markdown is available for agents
agent.markdown - Markdown is advertised and negotiated correctly
agent.markdown-negotiation
Why check llms.txt and Markdown together
llms.txt is a directory: a short Markdown file that tells a language model what your site is and
links to its most useful pages. The links work best when they point to Markdown versions of those
pages, so CrawlWise checks the file and the Markdown it leads to in the same scan.
The report shows:
- whether
/llms.txtanswers with200and a text content type, not a single-page app’s HTML shell that answers every path; - whether it starts with an H1 and lists at least three Markdown links, and whether an
llms-full.txtsits next to it; - whether the scanned page has a Markdown version, found through a
.mdURL, a<link rel="alternate" type="text/markdown">or anAccept: text/markdownrequest; - whether the Markdown names the HTML page as canonical and keeps the page’s text.
Common mistakes
- An HTML page at /llms.txt. Many frameworks answer every unknown path with the app shell. The file looks present in a browser but is not Markdown.
- No H1. It is the only required part of the file.
- Links to HTML pages full of navigation. Point to Markdown where you have it.
- Using llms.txt to block bots. It grants or denies nothing; use robots.txt for access.
Publish one in ten minutes
Write a Markdown file with your site name as the H1, a one-sentence summary in a blockquote and a
list of your key pages, and serve it at /llms.txt. The llms.txt guide has
an example, and Markdown for agents shows how to serve .md
versions with the right headers. CrawlWise publishes its own at
/llms.txt.
FAQ
What does an llms.txt validator check?
The proposal at llmstxt.org requires one thing, an H1 with the site name. It recommends a blockquote summary, then H2 sections of Markdown links with optional notes. CrawlWise checks that the file exists at the root, is not an HTML page, starts with an H1 and lists at least three Markdown links. It does not fetch the links.
Do ChatGPT or Google read llms.txt?
No major AI search engine has said it reads llms.txt. Some coding assistants and agent tools do. Treat it as a cheap, optional directory for agents, not as a ranking factor.
Is a missing llms.txt a problem?
Rarely. The check is experimental and worth little in the CrawlWise score. Crawler access and content that does not need JavaScript matter far more for AI search.
What is llms-full.txt?
A convention, not part of the proposal: the Markdown of every page in one file, so an agent can load a whole site in one request. It is useful for documentation sites.
Related guides
More free checkers
Last updated .