CrawlWise

Guides

Agent formats

Serve Markdown to AI agents: .md pages and headers

Give agents a clean Markdown version of each page: the URL convention, Accept: text/markdown, and the Link, Vary and canonical headers that keep it correct.

An HTML page carries a lot that an agent does not need: navigation, scripts, styles, cookie banners, footers. A Markdown version of the same page keeps what matters, the headings, text, lists, tables and links, in a format language models read well. Publishing one is a small change, but the headers around it decide whether it helps or creates a duplicate page in search.

Two ways to serve it

A separate URL. The llms.txt proposal suggests a Markdown version at the same URL with .md added: /pricing becomes /pricing.md, and the home page becomes /index.md. Any client can fetch it, and you can link to it from llms.txt.

Content negotiation. The page URL returns Markdown when the request says it prefers it:

GET /pricing HTTP/1.1
Accept: text/markdown, text/html

Cloudflare reports that coding agents such as Claude Code and OpenCode send this header. It is the more elegant option, but only clients that ask for Markdown get it.

You can do both from the same code. The rest of this guide shows the headers that make either option behave.

The headers that matter

Content-Type

Serve Markdown as text/markdown with a charset. The media type is registered in RFC 7763, and the charset parameter is required:

Content-Type: text/markdown; charset=utf-8

text/plain works for most clients, but text/markdown says what the content is.

Link to the Markdown from the HTML page

Tell agents the Markdown exists. In the HTML head:

<link rel="alternate" type="text/markdown" href="/pricing.md">

And in the HTTP response of the HTML page, so a client can find it without parsing HTML:

Link: <https://example.com/pricing.md>; rel="alternate"; type="text/markdown"

Point the Markdown back at the HTML page

Without this, a search engine can index /pricing.md as a second copy of /pricing. A Markdown file has no <head>, so use the HTTP header. Google supports rel="canonical" in a Link header for non-HTML documents, with an absolute URL:

Link: <https://example.com/pricing>; rel="canonical"

Vary and ETag when you negotiate

If one URL returns HTML or Markdown depending on Accept, add Vary: Accept. Otherwise a cache or CDN can store the Markdown and serve it to a browser, or the reverse. Give the two representations different ETag values too, so a conditional request for one is not answered with the other.

Content-Type: text/markdown; charset=utf-8
Vary: Accept
ETag: "pricing-v42-md"

What to put in the Markdown

  • The main content, not the chrome. Start with an H1 title, then the content in reading order. Leave out the navigation, cookie banner and footer.
  • The same content as the page. The Markdown should say what the HTML says. If the two drift apart, agents quote things your page no longer states.
  • Absolute links. A relative link such as [Pricing](/pricing) breaks when the Markdown is read outside your site, for example pasted into a chat. Write [Pricing](https://example.com/pricing).
  • Tables as tables. Markdown tables survive; layout made of <div> grids does not.

Generate the Markdown from the same source as the HTML wherever you can. A hand-written copy goes stale.

Examples

Rails

Register the type once, then answer both formats from the same action:

# config/initializers/mime_types.rb
Mime::Type.register "text/markdown", :markdown, [], %w[md]

# app/controllers/pages_controller.rb
def pricing
  respond_to do |format|
    format.html do
      response.headers["Link"] =
        %(<#{request.base_url}/pricing.md>; rel="alternate"; type="text/markdown")
    end
    format.markdown do
      response.headers["Link"] = %(<#{request.base_url}/pricing>; rel="canonical")
      render content_type: "text/markdown; charset=utf-8"
    end
  end
end

With a route such as get "pricing", to: "pages#pricing", Rails serves /pricing.md with pricing.markdown.erb. This is how the pages on crawlwise.dev are served; compare crawlwise.dev/methodology with crawlwise.dev/methodology.md.

A route handler

In any framework that uses the web Request and Response APIs, such as Next.js Route Handlers, Remix or Cloudflare Workers:

export async function GET(): Promise<Response> {
  const markdown = await renderPricingMarkdown();

  return new Response(markdown, {
    headers: {
      "Content-Type": "text/markdown; charset=utf-8",
      Link: '<https://example.com/pricing>; rel="canonical"',
      "Cache-Control": "public, max-age=300",
    },
  });
}

Cloudflare Markdown for Agents

Cloudflare can convert HTML to Markdown at its edge when a request sends Accept: text/markdown. It announced the feature in February 2026, and it is available on Pro, Business and Enterprise plans. You turn it on in AI Crawl Control, or per path with a Configuration Rule.

The converted response gets Vary: accept, and Cloudflare drops ETag and Last-Modified because conditional requests cannot be honoured for converted content. Note one default: if your origin sends no Content-Signal header, Cloudflare adds Content-Signal: ai-train=yes, search=yes, ai-input=yes. If you do not want to signal consent to training, send your own header from the origin.

Edge conversion is the fastest route, but it converts whatever the HTML contains. A Markdown view you render yourself can leave out the boilerplate and keep the structure you intend.

Check it

# The Markdown URL: status, type and canonical
curl -sS -D - -o /dev/null https://example.com/pricing.md | grep -iE '^(HTTP|content-type|link)'

# The HTML page advertises it
curl -sS -D - -o /dev/null https://example.com/pricing | grep -i '^link'

# Negotiation, if you support it
curl -sS -D - -o /dev/null -H 'Accept: text/markdown' https://example.com/pricing \
  | grep -iE '^(content-type|vary)'

A CrawlWise scan fetches the Markdown version of each audited page and checks its status, content type and H1, compares its text with the rendered page, and flags relative links. A second check confirms that the HTML advertises the Markdown, or that the response varies by Accept.

Sources

How CrawlWise checks this

Scan your page

Last updated .