Name: AI readiness of the top 100 e-commerce sites, 2026
Creator: CrawlWise
License: https://creativecommons.org/licenses/by/4.0/

Key numbers

Median score: 57.5/100
Block an AI search bot: 33%
Need JavaScript for content: 38%
Publish llms.txt: 6%

What we found

A third of the homepages keep AI search bots out. 18 of the 54 homepages where robots.txt could be checked block an AI search bot such as OAI-SearchBot or PerplexityBot; 24 block a training bot such as GPTBot or ClaudeBot. The two policies are separate decisions, and many shops make only one of them. The robots.txt guide covers the split.
Firewalls are making their own rules. 19 of 42 homepages answered an AI user agent differently from a regular request. A 403 at the edge overrides a permissive robots.txt, so the file alone is not proof that a bot gets in. Our note on Cloudflare and AI crawlers explains why.
The product facts are often behind JavaScript. 18 of 47 homepages needed JavaScript to show their main content. AI crawlers mostly do not run it, so the page they see can be a shell. Names, prices and stock are the facts a shopping answer needs. Do AI crawlers run JavaScript?
Almost no one publishes an agent-friendly version. 3 of 51 homepages have an llms.txt, and none of the 59 scored homepages serves a Markdown version. Both are optional, but they are the cheapest way to give an agent clean text. llms.txt and Markdown for agents explain when they help.
Structured data rarely describes the page. 12 of 59 homepages pass the core JSON-LD check, which asks for markup that parses and names the organisation, site or page. For a shop, product and price markup is what an answer can quote. The structured data guide covers the types worth adding.
The agent protocols are not there yet. None of the 59 scored homepages declared a Content-Signal or exposed WebMCP tools, and none published an API catalog, MCP server card or other agent discovery document. No homepage published DNS-AID records either. Most shops can ignore these; their absence says more about the state of the format than about any one site. The agent-readiness checklist sorts them by the kind of site that needs them.

How an agent reads a shop

An agent asks for a homepage, reads the response, and quotes what it can. Each step is a filter: a shop that fails one is less likely to appear in the answer.

The path from a request to an answer, with the share of assessed homepages at each step

Scores

The median homepage scored 57.5 out of 100 and the mean 58.9, from 40 to 78.1. The distribution is narrow: 32 of the 59 scored homepages sit between 50 and 59. No homepage reached an A (90 or more).

Homepages by CrawlWise score, in ten-point bands (59 scored)

Show the data as a table

Score	Homepages
0–9	0
10–19	0
20–29	0
30–39	0
40–49	7
50–59	32
60–69	10
70–79	10
80–89	0
90–100	0

Grade	A	B	C	D	E	F
Homepages	0	0	10	10	32	7

Where homepages lose points

These are the checks that fewest homepages passed. Weights and pass criteria are in the methodology.

Check	Weight	Passed
Markdown is available for agents	8	0%
sitemap.xml is valid and current	2	0%
llms.txt directory is present	2	3%
Document structure is extractable	5	8%
Automated accessibility checks pass	2	8%
Structured data describes the page entity	4	14%

What good looks like

Each row is a check the scanner runs, paired with the fix a CrawlWise report suggests. “Passed” is the share of assessed homepages that passed it here; a dash means the sample has no observation, and warnings count as partial passes.

Start at the top. Crawler access and server-rendered product text decide whether an agent can read the page at all; Markdown, llms.txt and the API documents only help after that.

Discovery
Content
Schema
Agent
Performance
Trust
Freshness & A11y
Agent and API protocols

Discovery

Can crawlers and AI agents find the page? Covers robots.txt, citation bot access and sitemap discovery.

Practice	Weight	Passed
robots.txt is reachable and valid Check the robots.txt URL and response in the evidence. Serve a valid plain-text policy at the site root; verify that redirects and access controls do not prevent crawlers from fetching it. Guide: AI crawlers in robots.txt: what to allow and block · `discovery.robots`	2	90%
AI search and citation bots are allowed Keep search and citation bots allowed. Decide training bots separately: blocking training is a policy choice, not an indexing error. Serve the policy from public/robots.txt in Rails or a robots route in Next.js. Guide: AI crawlers in robots.txt: what to allow and block · `discovery.ai-bots`	6	69%
AI crawlers get the same page as a regular fetch Check the firewall, CDN and bot management rules for the bots in the evidence. Allow AI search and user-triggered crawlers there as well as in robots.txt; in Cloudflare, review AI Crawl Control and the AI Crawler and Agent access policies. Guide: Is Cloudflare blocking AI crawlers on your site? · `discovery.ai-bot-access`	2	55%
sitemap.xml is valid and current Check the sitemap URL and errors in the evidence. Publish a valid XML sitemap containing canonical public URLs, and reference it from robots.txt. `discovery.sitemap`	2	0%
The page is indexable and has a consistent canonical Remove noindex, none, nofollow or nosnippet directives that block indexing or snippets. Add a self-referencing canonical that matches the final URL. `discovery.indexability`	8	66%
Redirects are permanent and short Use one permanent 301 or 308 redirect to the final HTTPS URL. Remove temporary and chained hops, and update internal links to the destination. `discovery.redirects`	3	75%
robots.txt allows CSS and JavaScript needed for rendering Do not disallow CSS or JavaScript paths that Googlebot needs to render the page. Check the rules for Googlebot and the catch-all group, and keep render-critical assets crawlable. Guide: AI crawlers in robots.txt: what to allow and block · `discovery.resource-access`	3	90%
hreflang annotations are complete Add a self-referencing hreflang entry for each language, one x-default, and keep language codes unique. Verify that every alternate links back. `discovery.hreflang`	1	86%

Content

What a visitor or agent gets without JavaScript: main text, page structure, boilerplate and consent walls.

Practice	Weight	Passed
Main content is available without JavaScript Compare the raw and rendered content in the evidence. Render the important page text in the initial HTML: use server-rendered views in Rails or server rendering/static generation in Next.js. Guide: Do AI crawlers run JavaScript? Mostly, no · `content.no-js`	7	55%
Document structure is extractable Review the extracted content in the evidence. Put the primary content in a main element with descriptive headings and keep navigation separate from the article or page body. `content.structure`	5	8%
Main content is not buried in boilerplate Reduce repeated navigation, footer and promotional blocks. Keep the primary content inside main or article and remove duplicated copy from the template. `content.boilerplate`	1	20%
Content is not hidden behind a consent wall Keep the primary content in the initial HTML. If you use a consent banner, do not let it remove or hide the text from crawlers that do not run JavaScript. Guide: Do AI crawlers run JavaScript? Mostly, no · `content.consent-wall`	1	93%
Content images are described and efficient Add meaningful alt text, explicit width and height, and serve WebP or AVIF where the browser supports it. Keep decorative images with empty alt text. `content.images`	3	22%
Internal links are crawlable and descriptive Use crawlable a href links with descriptive anchor text. Avoid nofollow on internal navigation, replace generic anchors such as click here, and link to your own HTTPS URLs instead of HTTP. `content.links`	2	20%
Social preview metadata is complete Add og:title, og:description and an absolute og:image (1200x630 is a safe size) to every public page. Add twitter:card and twitter:image if you want a large card on X. `content.social`	1	22%

Schema

Structured data: JSON-LD parses, core fields are present and dates are valid.

Practice	Weight	Passed
JSON-LD parses and describes the page Inspect the JSON-LD errors and missing fields in the evidence. Correct the JSON syntax and add accurate schema.org types and properties that describe the visible page. Guide: Structured data for AI search: what helps and what doesn't · `schema.parses-core`	8	20%
Structured data dates are valid Use valid ISO 8601 datePublished and dateModified values, keep dateModified at or after datePublished, and expose a visible update date. Guide: Structured data for AI search: what helps and what doesn't · `schema.dates`	3	100%
Structured data describes the page entity Add a page-level entity using Article, Product, FAQPage or HowTo as appropriate, with author, publisher and sameAs. Add BreadcrumbList for non-home pages. Guide: Structured data for AI search: what helps and what doesn't · `schema.entities`	4	14%

Agent

Agent-specific signals: a Markdown version of the page and an llms.txt file.

Practice	Weight	Passed
Markdown is available for agents If a Markdown representation is useful for your audience, provide one containing the same primary content as the HTML and link to it from the page. Keep both versions in sync. Guide: llms.txt: what it is and whether you need one · `agent.markdown`	8	0%
llms.txt directory is present Optional: publish llms.txt and llms-full.txt with stable Markdown links to your best content. Do not treat them as access control or a search ranking signal. Guide: llms.txt: what it is and whether you need one · `agent.llms-txt`	2	3%
Markdown is advertised and negotiated correctly Serve the Markdown representation as text/markdown, advertise it with a rel=alternate link, and send Vary: Accept when content negotiation is used. Guide: Serve Markdown to AI agents: .md pages and headers · `agent.markdown-negotiation`	2	0%

Performance

Fetch and render performance: TTFB, HTML payload, caching, resource budget and lab LCP/CLS.

Practice	Weight	Passed
Time to first byte is reasonable Cache HTML at the edge, reduce server and database work, and remove slow upstream calls from the first response. Measure from the scan region. `perf.ttfb`	2	86%
HTML payload is small and compressed Compress HTML with Brotli or gzip, remove unused inline bloat, and serve the page over HTTP/2 or HTTP/3. `perf.payload`	2	53%
HTML has validators and cache headers Set Cache-Control and ETag or Last-Modified for public HTML. Avoid no-store on pages that agents and browsers should reuse. `perf.caching`	2	24%
Page resources stay within a reasonable budget Inspect the largest resources and blocking scripts in the evidence. Resize and compress oversized images, remove unused assets, and defer scripts that are not needed for the initial render. `perf.resources`	3	14%
Largest Contentful Paint is under 2.5 seconds Inspect the LCP element and URL in the evidence. Prioritize the main image or text, reduce render-blocking work, and investigate slow server responses. Re-test under representative conditions. `perf.lcp`	3	78%
Cumulative Layout Shift is under 0.1 Inspect the shifting elements in the evidence. Reserve space for images and embeds, avoid inserting content above existing content, and check font loading. Re-test with real user data when available. `perf.cls`	3	85%

Trust

Security and trust: HTTPS, HSTS, security headers and redirect behaviour.

Practice	Weight	Passed
HTTPS, HSTS and redirects are correct Redirect HTTP to HTTPS in one permanent hop, add an HSTS policy, and remove mixed-content references from the rendered page. `trust.https`	6	86%
Security headers are present Add a Content-Security-Policy, X-Content-Type-Options: nosniff, a Referrer-Policy and frame protection. `trust.headers`	2	17%

Machine-readable dates and automated accessibility checks.

Practice	Weight	Passed
Freshness signals are present Expose valid dateModified in structured data and as a visible update date, and keep the sitemap lastmod consistent. `freshness.signals`	1	15%
Automated accessibility checks pass Fix critical and serious axe violations first. Open the evidence rules and address the affected elements before minor issues. Guide: Lighthouse Agentic Browsing: what it checks · `a11y.axe`	2	8%

Agent and API protocols

These checks carry no weight: a shop without an API or an agent can ignore them. They are listed because an agent that wants to search, compare or buy can use them when they exist.

Practice	Weight	Passed
Content usage preferences for AI are declared Optional: state how AI systems may use your content with a Content-Signal line in robots.txt, for example search=yes, ai-input=yes, ai-train=no. Use only the keys search, ai-input and ai-train with yes or no. Guide: Is Cloudflare blocking AI crawlers on your site? · `discovery.content-signal`	—	0%
WebMCP tools are complete Give every WebMCP tool a name, a description and an input schema: registerTool() needs all three plus execute, and a form needs both toolname and tooldescription. WebMCP is an origin trial, so keep the same actions available without it. Guide: WebMCP: let AI agents use your site's forms · `api.webmcp`	—	0%
An API catalog lists the site's APIs Serve /.well-known/api-catalog as application/linkset+json with a linkset array; give each API an anchor and links to its OpenAPI description (service-desc) and documentation (service-doc). See RFC 9727. Guide: Is your site agent-ready? A practical checklist · `api.catalog`	—	0%
OAuth or OpenID Connect discovery is published Publish /.well-known/oauth-authorization-server (RFC 8414) or /.well-known/openid-configuration with the issuer and the authorization and token endpoints, so agents can find how to get a token. Guide: Is your site agent-ready? A practical checklist · `api.oauth-discovery`	—	0%
OAuth protected resource metadata is published Publish /.well-known/oauth-protected-resource (RFC 9728) with the resource identifier and the authorization servers that issue its tokens. Guide: Is your site agent-ready? A practical checklist · `api.oauth-protected-resource`	—	0%
An MCP server card describes the site's MCP server Serve /.well-known/mcp/server-card.json with serverInfo (name and version), the transport endpoint and the server's capabilities. The format is a draft, so check the current MCP proposal. Guide: Is your site agent-ready? A practical checklist · `api.mcp-server-card`	—	0%
An A2A agent card describes the site's agent Serve /.well-known/agent-card.json with the agent's name, description, the interfaces that reach it and its skills, as the A2A specification describes. Guide: Is your site agent-ready? A practical checklist · `api.a2a-agent-card`	—	0%
An agent skills index lists the site's skills Publish /.well-known/agent-skills/index.json with the 0.2.0 $schema and a skills array; give each skill a lowercase hyphenated name, a type (skill-md or archive), a description, a url and a sha256 digest. Guide: Is your site agent-ready? A practical checklist · `api.agent-skills`	—	0%
An ARD manifest lists the site's agent resources Serve /.well-known/ard.json with an entries array. Give each entry a urn:air:<your-domain>:<name> identifier, a displayName, a media type in type, exactly one of url or data, and 2 to 5 representativeQueries. Move a manifest from /.well-known/ai-catalog.json to the new path. Guide: Is your site agent-ready? A practical checklist · `api.ard`	—	0%

Method and limits

The sample is OneLittleWeb's top 100 e-commerce websites for September 2026, ranked by estimated traffic. It counts websites, not companies, so marketplaces, classifieds, price comparison, logistics and other non-retail services sit next to 14 Amazon storefronts. We kept the list as published.
Each homepage was scanned once from one machine in Europe with the standard CrawlWise checkset. 59 could be scored; 34 returned a block or challenge, 6 failed, and 1 opted out through robots.txt. Those sites are outside the check denominators, which is why every share says how many homepages it covers.
These are homepages, not shops. A product page, a category page or the page an agent reaches from a search result can score differently. A sitemap that lists products but not the homepage, or one behind a bot challenge, is reported as missing even when the shop publishes one.
AI user agents were sent from our address, not from the crawler operators' addresses. A firewall that verifies IPs can treat the real crawlers differently.
Markdown counts when the homepage advertises it or serves it for Accept: text/markdown. Word overlap with the rendered page measures whether it matches; navigation and link URLs in converted Markdown lower that score.
Performance figures come from one unthrottled browser visit. They are lab observations, not field Core Web Vitals or PageSpeed Insights scores. The traffic numbers in the ranking are estimates, not audited visits.

Data

The per-check counts, with no site named: CSV · JSON.

The aggregate CSV and JSON are available under the Creative Commons Attribution 4.0 International license. The article text and the CrawlWise name are not covered by that license.

Cite this study: CrawlWise, “AI readiness of the top 100 e-commerce sites” (2026), https://crawlwise.dev/research/ecommerce-100-ai-readiness-2026.