# AI readiness of the top 100 e-commerce sites [OneLittleWeb ranks e-commerce websites by estimated traffic](https://onelittleweb.com/digital-market-intelligence/popular-websites/e-commerce/). We took its September 2026 top 100 and scanned each homepage once with CrawlWise, asking what an AI crawler or agent would actually get: permission to fetch, product text without JavaScript, structured data, and a Markdown or `llms.txt` version. 59 of the 100 homepages could be scored. Scanned on 6 November 2026 with checkset 0.4.10. Aggregates only: no site is named. The ranking counts websites, not companies, so marketplaces, classifieds, price comparison, logistics and other non-retail services sit alongside 14 Amazon storefronts. ## Key numbers - Median score: 57.5 of 100 (mean 58.9, range 40–78.1) - Block an AI search bot in robots.txt: 18 of 54 (33%) - Block AI training bots in robots.txt: 24 of 54 (44%) - Answer AI user agents differently: 19 of 42 (45%) - Need JavaScript for the main content: 18 of 47 (38%) - Publish llms.txt: 3 of 51 (6%) - Serve a Markdown version: 0 of 59 (0%) - JSON-LD that describes the organisation and page: 12 of 59 (20%) - Declare a Content-Signal: 0 of 59 (0%) - Publish an API or agent discovery document: 0 of 59 (0%) - Expose WebMCP tools: 0 of 59 (0%) - Publish DNS-AID records: 0 of 100 (0%) ## How an agent reads a shop 1. An AI agent requests a homepage: 100 in the sample, 59 could be scored. 2. `robots.txt`: 18 of 54 (33%) block an AI search bot; 24 of 54 (44%) block a training bot. 3. HTML response: 18 of 47 (38%) need JavaScript for the main content. 4. Structured data: 12 of 59 (20%) pass core JSON-LD. 5. Agent formats: 3 of 51 (6%) publish llms.txt; 0 of 59 (0%) serve a Markdown version. 6. Answer or citation: what the agent can read, quote and link back to. ## Scores The median homepage scored 57.5 out of 100 and the mean 58.9, from 40 to 78.1. The distribution is narrow: 32 of the 59 scored homepages sit between 50 and 59. No homepage reached an A (90 or more). | Score | Homepages | |---|---| | 0–9 | 0 | | 10–19 | 0 | | 20–29 | 0 | | 30–39 | 0 | | 40–49 | 7 | | 50–59 | 32 | | 60–69 | 10 | | 70–79 | 10 | | 80–89 | 0 | | 90–100 | 0 | | Grade | A | B | C | D | E | F | |---|---|---|---|---|---|---| | Homepages | 0 | 0 | 10 | 10 | 32 | 7 | ## Where homepages lose points | Check | Weight | Passed | |---|---|---| | Markdown is available for agents | 8 | 0% | | sitemap.xml is valid and current | 2 | 0% | | llms.txt directory is present | 2 | 3% | | Document structure is extractable | 5 | 8% | | Automated accessibility checks pass | 2 | 8% | | Structured data describes the page entity | 4 | 14% | ## What good looks like Each row is a check the scanner runs, paired with the fix a CrawlWise report suggests. "Passed" is the share of assessed homepages that passed it here; a dash means the sample has no observation, and warnings count as partial passes. Start at the top. Crawler access and server-rendered product text decide whether an agent can read the page at all; Markdown, `llms.txt` and the API documents only help after that. ### Discovery Can crawlers and AI agents find the page? Covers robots.txt, citation bot access and sitemap discovery. - **robots.txt is reachable and valid** (weight 2, passed 90%). Check the robots.txt URL and response in the evidence. Serve a valid plain-text policy at the site root; verify that redirects and access controls do not prevent crawlers from fetching it. Guide: . - **AI search and citation bots are allowed** (weight 6, passed 69%). Keep search and citation bots allowed. Decide training bots separately: blocking training is a policy choice, not an indexing error. Serve the policy from public/robots.txt in Rails or a robots route in Next.js. Guide: . - **AI crawlers get the same page as a regular fetch** (weight 2, passed 55%). Check the firewall, CDN and bot management rules for the bots in the evidence. Allow AI search and user-triggered crawlers there as well as in robots.txt; in Cloudflare, review AI Crawl Control and the AI Crawler and Agent access policies. Guide: . - **sitemap.xml is valid and current** (weight 2, passed 0%). Check the sitemap URL and errors in the evidence. Publish a valid XML sitemap containing canonical public URLs, and reference it from robots.txt. - **The page is indexable and has a consistent canonical** (weight 8, passed 66%). Remove noindex, none, nofollow or nosnippet directives that block indexing or snippets. Add a self-referencing canonical that matches the final URL. - **Redirects are permanent and short** (weight 3, passed 75%). Use one permanent 301 or 308 redirect to the final HTTPS URL. Remove temporary and chained hops, and update internal links to the destination. - **robots.txt allows CSS and JavaScript needed for rendering** (weight 3, passed 90%). Do not disallow CSS or JavaScript paths that Googlebot needs to render the page. Check the rules for Googlebot and the catch-all group, and keep render-critical assets crawlable. Guide: . - **hreflang annotations are complete** (weight 1, passed 86%). Add a self-referencing hreflang entry for each language, one x-default, and keep language codes unique. Verify that every alternate links back. ### Content What a visitor or agent gets without JavaScript: main text, page structure, boilerplate and consent walls. - **Main content is available without JavaScript** (weight 7, passed 55%). Compare the raw and rendered content in the evidence. Render the important page text in the initial HTML: use server-rendered views in Rails or server rendering/static generation in Next.js. Guide: . - **Document structure is extractable** (weight 5, passed 8%). Review the extracted content in the evidence. Put the primary content in a main element with descriptive headings and keep navigation separate from the article or page body. - **Main content is not buried in boilerplate** (weight 1, passed 20%). Reduce repeated navigation, footer and promotional blocks. Keep the primary content inside main or article and remove duplicated copy from the template. - **Content is not hidden behind a consent wall** (weight 1, passed 93%). Keep the primary content in the initial HTML. If you use a consent banner, do not let it remove or hide the text from crawlers that do not run JavaScript. Guide: . - **Content images are described and efficient** (weight 3, passed 22%). Add meaningful alt text, explicit width and height, and serve WebP or AVIF where the browser supports it. Keep decorative images with empty alt text. - **Internal links are crawlable and descriptive** (weight 2, passed 20%). Use crawlable a href links with descriptive anchor text. Avoid nofollow on internal navigation, replace generic anchors such as click here, and link to your own HTTPS URLs instead of HTTP. - **Social preview metadata is complete** (weight 1, passed 22%). Add og:title, og:description and an absolute og:image (1200x630 is a safe size) to every public page. Add twitter:card and twitter:image if you want a large card on X. ### Schema Structured data: JSON-LD parses, core fields are present and dates are valid. - **JSON-LD parses and describes the page** (weight 8, passed 20%). Inspect the JSON-LD errors and missing fields in the evidence. Correct the JSON syntax and add accurate schema.org types and properties that describe the visible page. Guide: . - **Structured data dates are valid** (weight 3, passed 100%). Use valid ISO 8601 datePublished and dateModified values, keep dateModified at or after datePublished, and expose a visible update date. Guide: . - **Structured data describes the page entity** (weight 4, passed 14%). Add a page-level entity using Article, Product, FAQPage or HowTo as appropriate, with author, publisher and sameAs. Add BreadcrumbList for non-home pages. Guide: . ### Agent Agent-specific signals: a Markdown version of the page and an llms.txt file. - **Markdown is available for agents** (weight 8, passed 0%). If a Markdown representation is useful for your audience, provide one containing the same primary content as the HTML and link to it from the page. Keep both versions in sync. Guide: . - **llms.txt directory is present** (weight 2, passed 3%). Optional: publish llms.txt and llms-full.txt with stable Markdown links to your best content. Do not treat them as access control or a search ranking signal. Guide: . - **Markdown is advertised and negotiated correctly** (weight 2, passed 0%). Serve the Markdown representation as text/markdown, advertise it with a rel=alternate link, and send Vary: Accept when content negotiation is used. Guide: . ### Performance Fetch and render performance: TTFB, HTML payload, caching, resource budget and lab LCP/CLS. - **Time to first byte is reasonable** (weight 2, passed 86%). Cache HTML at the edge, reduce server and database work, and remove slow upstream calls from the first response. Measure from the scan region. - **HTML payload is small and compressed** (weight 2, passed 53%). Compress HTML with Brotli or gzip, remove unused inline bloat, and serve the page over HTTP/2 or HTTP/3. - **HTML has validators and cache headers** (weight 2, passed 24%). Set Cache-Control and ETag or Last-Modified for public HTML. Avoid no-store on pages that agents and browsers should reuse. - **Page resources stay within a reasonable budget** (weight 3, passed 14%). Inspect the largest resources and blocking scripts in the evidence. Resize and compress oversized images, remove unused assets, and defer scripts that are not needed for the initial render. - **Largest Contentful Paint is under 2.5 seconds** (weight 3, passed 78%). Inspect the LCP element and URL in the evidence. Prioritize the main image or text, reduce render-blocking work, and investigate slow server responses. Re-test under representative conditions. - **Cumulative Layout Shift is under 0.1** (weight 3, passed 85%). Inspect the shifting elements in the evidence. Reserve space for images and embeds, avoid inserting content above existing content, and check font loading. Re-test with real user data when available. ### Trust Security and trust: HTTPS, HSTS, security headers and redirect behaviour. - **HTTPS, HSTS and redirects are correct** (weight 6, passed 86%). Redirect HTTP to HTTPS in one permanent hop, add an HSTS policy, and remove mixed-content references from the rendered page. - **Security headers are present** (weight 2, passed 17%). Add a Content-Security-Policy, X-Content-Type-Options: nosniff, a Referrer-Policy and frame protection. ### Freshness & A11y Machine-readable dates and automated accessibility checks. - **Freshness signals are present** (weight 1, passed 15%). Expose valid dateModified in structured data and as a visible update date, and keep the sitemap lastmod consistent. - **Automated accessibility checks pass** (weight 2, passed 8%). Fix critical and serious axe violations first. Open the evidence rules and address the affected elements before minor issues. Guide: . ### Agent and API protocols These checks carry no weight: a shop without an API or an agent can ignore them. They are listed because an agent that wants to search, compare or buy can use them when they exist. - **Content usage preferences for AI are declared** (not scored, passed 0%). Optional: state how AI systems may use your content with a Content-Signal line in robots.txt, for example search=yes, ai-input=yes, ai-train=no. Use only the keys search, ai-input and ai-train with yes or no. Guide: . - **WebMCP tools are complete** (not scored, passed 0%). Give every WebMCP tool a name, a description and an input schema: registerTool() needs all three plus execute, and a form needs both toolname and tooldescription. WebMCP is an origin trial, so keep the same actions available without it. Guide: . - **An API catalog lists the site's APIs** (not scored, passed 0%). Serve /.well-known/api-catalog as application/linkset+json with a linkset array; give each API an anchor and links to its OpenAPI description (service-desc) and documentation (service-doc). See RFC 9727. Guide: . - **OAuth or OpenID Connect discovery is published** (not scored, passed 0%). Publish /.well-known/oauth-authorization-server (RFC 8414) or /.well-known/openid-configuration with the issuer and the authorization and token endpoints, so agents can find how to get a token. Guide: . - **OAuth protected resource metadata is published** (not scored, passed 0%). Publish /.well-known/oauth-protected-resource (RFC 9728) with the resource identifier and the authorization servers that issue its tokens. Guide: . - **An MCP server card describes the site's MCP server** (not scored, passed 0%). Serve /.well-known/mcp/server-card.json with serverInfo (name and version), the transport endpoint and the server's capabilities. The format is a draft, so check the current MCP proposal. Guide: . - **An A2A agent card describes the site's agent** (not scored, passed 0%). Serve /.well-known/agent-card.json with the agent's name, description, the interfaces that reach it and its skills, as the A2A specification describes. Guide: . - **An agent skills index lists the site's skills** (not scored, passed 0%). Publish /.well-known/agent-skills/index.json with the 0.2.0 $schema and a skills array; give each skill a lowercase hyphenated name, a type (skill-md or archive), a description, a url and a sha256 digest. Guide: . - **An ARD manifest lists the site's agent resources** (not scored, passed 0%). Serve /.well-known/ard.json with an entries array. Give each entry a urn:air:: identifier, a displayName, a media type in type, exactly one of url or data, and 2 to 5 representativeQueries. Move a manifest from /.well-known/ai-catalog.json to the new path. Guide: . ## Method and limits - The sample is OneLittleWeb's top 100 e-commerce websites for September 2026, ranked by estimated traffic. It counts websites, not companies, so marketplaces, classifieds, price comparison, logistics and other non-retail services sit next to 14 Amazon storefronts. We kept the list as published. - Each homepage was scanned once from one machine in Europe with the standard CrawlWise checkset: . 59 could be scored; 34 returned a block or challenge, 6 failed, and 1 opted out through robots.txt. Those sites are outside the check denominators, which is why every share says how many homepages it covers. - These are homepages, not shops. A product page, a category page or the page an agent reaches from a search result can score differently. A sitemap that lists products but not the homepage, or one behind a bot challenge, is reported as missing even when the shop publishes one. - AI user agents were sent from our address, not from the crawler operators' addresses. A firewall that verifies IPs can treat the real crawlers differently. - Markdown counts when the homepage advertises it or serves it for `Accept: text/markdown`. Word overlap with the rendered page measures whether it matches; navigation and link URLs in converted Markdown lower that score. - Performance figures come from one unthrottled browser visit. They are lab observations, not field Core Web Vitals or PageSpeed Insights scores. The traffic numbers in the ranking are estimates, not audited visits. ## Data The per-check counts, with no site named. - CSV: - JSON: - License: the aggregate CSV and JSON are available under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). The article text and the CrawlWise name are not covered by that license. - Cite this study: CrawlWise, "AI readiness of the top 100 e-commerce sites" (2026), . Scan your own shop: --- Last updated 6 November 2026. HTML version: Every CrawlWise page in Markdown: