# Serve Markdown to AI agents: .md pages and headers

An HTML page carries a lot that an agent does not need: navigation, scripts, styles, cookie
banners, footers. A Markdown version of the same page keeps what matters, the headings, text,
lists, tables and links, in a format language models read well. Publishing one is a small change,
but the headers around it decide whether it helps or creates a duplicate page in search.

## Two ways to serve it

**A separate URL.** The [llms.txt proposal](/guides/llms-txt) suggests a Markdown version at the
same URL with `.md` added: `/pricing` becomes `/pricing.md`, and the home page becomes
`/index.md`. Any client can fetch it, and you can link to it from `llms.txt`.

**Content negotiation.** The page URL returns Markdown when the request says it prefers it:

```http
GET /pricing HTTP/1.1
Accept: text/markdown, text/html
```

Cloudflare reports that coding agents such as Claude Code and OpenCode send this header. It is
the more elegant option, but only clients that ask for Markdown get it.

You can do both from the same code. The rest of this guide shows the headers that make either
option behave.

## The headers that matter

### Content-Type

Serve Markdown as `text/markdown` with a charset. The media type is registered in RFC 7763, and
the charset parameter is required:

```http
Content-Type: text/markdown; charset=utf-8
```

`text/plain` works for most clients, but `text/markdown` says what the content is.

### Link to the Markdown from the HTML page

Tell agents the Markdown exists. In the HTML head:

```html
<link rel="alternate" type="text/markdown" href="/pricing.md">
```

And in the HTTP response of the HTML page, so a client can find it without parsing HTML:

```http
Link: <https://example.com/pricing.md>; rel="alternate"; type="text/markdown"
```

### Point the Markdown back at the HTML page

Without this, a search engine can index `/pricing.md` as a second copy of `/pricing`. A Markdown
file has no `<head>`, so use the HTTP header. Google supports `rel="canonical"` in a `Link` header
for non-HTML documents, with an absolute URL:

```http
Link: <https://example.com/pricing>; rel="canonical"
```

### Vary and ETag when you negotiate

If one URL returns HTML or Markdown depending on `Accept`, add `Vary: Accept`. Otherwise a cache
or CDN can store the Markdown and serve it to a browser, or the reverse. Give the two
representations different `ETag` values too, so a conditional request for one is not answered
with the other.

```http
Content-Type: text/markdown; charset=utf-8
Vary: Accept
ETag: "pricing-v42-md"
```

## What to put in the Markdown

- **The main content, not the chrome.** Start with an H1 title, then the content in reading order.
  Leave out the navigation, cookie banner and footer.
- **The same content as the page.** The Markdown should say what the HTML says. If the two drift
  apart, agents quote things your page no longer states.
- **Absolute links.** A relative link such as `[Pricing](/pricing)` breaks when the Markdown is
  read outside your site, for example pasted into a chat. Write
  `[Pricing](https://example.com/pricing)`.
- **Tables as tables.** Markdown tables survive; layout made of `<div>` grids does not.

Generate the Markdown from the same source as the HTML wherever you can. A hand-written copy goes
stale.

## Examples

### Rails

Register the type once, then answer both formats from the same action:

```ruby
# config/initializers/mime_types.rb
Mime::Type.register "text/markdown", :markdown, [], %w[md]

# app/controllers/pages_controller.rb
def pricing
  respond_to do |format|
    format.html do
      response.headers["Link"] =
        %(<#{request.base_url}/pricing.md>; rel="alternate"; type="text/markdown")
    end
    format.markdown do
      response.headers["Link"] = %(<#{request.base_url}/pricing>; rel="canonical")
      render content_type: "text/markdown; charset=utf-8"
    end
  end
end
```

With a route such as `get "pricing", to: "pages#pricing"`, Rails serves `/pricing.md` with
`pricing.markdown.erb`. This is how the pages on crawlwise.dev are served; compare
[crawlwise.dev/methodology](https://crawlwise.dev/methodology) with
[crawlwise.dev/methodology.md](https://crawlwise.dev/methodology.md).

### A route handler

In any framework that uses the web `Request` and `Response` APIs, such as Next.js Route Handlers,
Remix or Cloudflare Workers:

```ts
export async function GET(): Promise<Response> {
  const markdown = await renderPricingMarkdown();

  return new Response(markdown, {
    headers: {
      "Content-Type": "text/markdown; charset=utf-8",
      Link: '<https://example.com/pricing>; rel="canonical"',
      "Cache-Control": "public, max-age=300",
    },
  });
}
```

### Cloudflare Markdown for Agents

Cloudflare can convert HTML to Markdown at its edge when a request sends `Accept: text/markdown`.
It announced the feature in February 2026, and it is available on Pro, Business and Enterprise
plans. You turn it on in **AI Crawl Control**, or per path with a Configuration Rule.

The converted response gets `Vary: accept`, and Cloudflare drops `ETag` and `Last-Modified`
because conditional requests cannot be honoured for converted content. Note one default: if your
origin sends no `Content-Signal` header, Cloudflare adds
`Content-Signal: ai-train=yes, search=yes, ai-input=yes`. If you do not want to signal consent to
training, send your own header from the origin.

Edge conversion is the fastest route, but it converts whatever the HTML contains. A Markdown view
you render yourself can leave out the boilerplate and keep the structure you intend.

## Check it

```sh
# The Markdown URL: status, type and canonical
curl -sS -D - -o /dev/null https://example.com/pricing.md | grep -iE '^(HTTP|content-type|link)'

# The HTML page advertises it
curl -sS -D - -o /dev/null https://example.com/pricing | grep -i '^link'

# Negotiation, if you support it
curl -sS -D - -o /dev/null -H 'Accept: text/markdown' https://example.com/pricing \
  | grep -iE '^(content-type|vary)'
```

A [CrawlWise scan](/) fetches the Markdown version of each audited page and checks its status,
content type and H1, compares its text with the rendered page, and flags relative links. A second
check confirms that the HTML advertises the Markdown, or that the response varies by `Accept`.

## Sources

- Jeremy Howard: [The /llms.txt file](https://llmstxt.org/)
- IETF: [RFC 7763, The text/markdown Media Type](https://datatracker.ietf.org/doc/html/rfc7763)
- IETF: [RFC 8288, Web Linking](https://www.rfc-editor.org/rfc/rfc8288)
- IETF: [RFC 9110, HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110)
- Google: [How to specify a canonical URL with rel="canonical" and other methods](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls)
- Cloudflare: [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/)
- Cloudflare: [Introducing Markdown for Agents](https://blog.cloudflare.com/markdown-for-agents/)

---

Last updated 3 November 2026.

HTML version: <https://crawlwise.dev/guides/markdown-for-agents>

Every CrawlWise page in Markdown: <https://crawlwise.dev/llms.txt>