Is your site agent-ready? A practical checklist

“Agent-ready” has become a score you can get from several tools, and the scores disagree. A company website can pass every check that matters for being found and cited by AI assistants and still score low, because the same checklist also covers OAuth servers, MCP endpoints and payment protocols that only an API or an online shop would publish.

This checklist sorts the signals by the kind of site that needs them. Start at the top; stop where your site stops.

Who the agents are

Three kinds of software read your site on behalf of AI products:

• Crawlers collect pages for search indexes and model training: OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot.
• User-triggered fetchers load one page because a person asked about it: ChatGPT-User, Claude-User, Perplexity-User.
• Browser agents operate a real browser for a person: they read the page, click, and fill in forms.

Most of them do not run JavaScript, and none of them get past a login. What they get is what your server sends.

Every site: can agents get in and read the page?

These decide whether you appear in AI answers at all. They matter for a one-page brochure site as much as for a SaaS product.

1. robots.txt lets AI search bots in. Allow the search and user-triggered bots; decide on training bots separately. See AI crawlers in robots.txt.
2. The firewall agrees with robots.txt. A CDN or bot manager that returns 403 to GPTBot overrides the file. See Is Cloudflare blocking AI crawlers on your site?.
3. The content is in the HTML. The main text, prices and facts arrive in the server response, not after JavaScript runs. See Do AI crawlers run JavaScript?.
4. The page is indexable. No noindex, a canonical URL that points to itself, a sitemap that lists the page.
5. The structure is clear. One h1, descriptive headings, real links, text not baked into images.
6. Structured data describes the page. An Organization, the page type and breadcrumbs in JSON-LD. See Structured data for AI search.
7. The page works for assistive technology. Browser agents read the accessibility tree, so labelled buttons and form fields help them the way they help screen readers.

Content sites: make the content easy to take in

These are cheap, help agents that already found you, and cost nothing in search.

1. A Markdown version of each page, linked from the HTML, and served for Accept: text/markdown. See Serve Markdown to AI agents.
2. An llms.txt index of your most useful pages. Useful for documentation and products people ask assistants about. See llms.txt: what it is.
3. A content usage preference in robots.txt, such as Content-Signal: search=yes, ai-input=yes, ai-train=no. It states how fetched content may be used; it does not block anything.
4. Link headers on the home page that point to the Markdown version and, if you have one, the API catalog.

Sites with forms or actions: let browser agents act

If people book, buy, search or submit something on your site, a browser agent will try to do it for them.

• Plain HTML forms with labels work for agents today: every field with a name and a <label>, buttons with text.
• WebMCP lets a page declare those actions as tools an agent can call, either with attributes on a <form> or with document.modelContext.registerTool(). It is an origin trial in Chrome. See WebMCP: let AI agents use your forms.

Sites with an API or an app: publish discovery documents

This part is for products that agents use programmatically. A content site can skip it.

Publish only what describes something real. An empty MCP card or a catalog that points at nothing is worse than none.

What the readiness scores measure

Two tools put a number on agent readiness, and both are worth knowing:

• Cloudflare’s Agent Readiness score (isitagentready.com) checks robots.txt, sitemaps, Link headers, Markdown negotiation, Content Signals and the API, auth, MCP and commerce documents above. It has profiles: the content-site profile leaves out the API and commerce checks, so use it for a company website. That profile still counts DNS for AI Discovery (DNS-AID), SVCB records at _index._agents.example.com that point to an organisation’s agent index. DNS-AID is an individual IETF draft, and a site without agents has nothing to publish there.
• Lighthouse’s Agentic Browsing category checks the accessibility tree, WebMCP, layout stability, llms.txt and an agent resource manifest. See Lighthouse Agentic Browsing.

Adoption is still low. When Cloudflare scanned the 200,000 most visited domains for its launch in April 2026, 78% had a robots.txt, 4% declared Content Signals, 3.9% handled Markdown negotiation, and MCP server cards and API catalogs together appeared on fewer than 15 sites.

That is the opportunity: the basics in the first two sections already put a site ahead of most of the web. Our study of the Forbes Cloud 100 shows the same pattern among the largest cloud companies, and the e-commerce 100 study finds it among the most-visited shops.

Check your site

A CrawlWise scan checks the items on this list, from crawler access and the firewall to the discovery documents. The report scores the first two sections and shows WebMCP and the API and agent protocol documents in a separate, unscored tab, so a content site is not marked down for not running an OAuth server.

Sources

• Cloudflare: Introducing the Agent Readiness score
• Chrome: Lighthouse Agentic Browsing
• Chrome: WebMCP
• IETF: RFC 9727, api-catalog
• IETF: RFC 8414, OAuth 2.0 Authorization Server Metadata
• IETF: RFC 9728, OAuth 2.0 Protected Resource Metadata
• A2A: Agent discovery
• Cloudflare: Agent Skills Discovery via well-known URIs
• Cloudflare: Content Signals