AI readiness of the Forbes Cloud 100

Key numbers

Median score

77.6/100

Publish llms.txt

65%

Serve a Markdown version

24%

Need JavaScript for content

7%

What we found

• The robots.txt door is open. None of the 82 homepages where robots.txt could be checked blocks an AI search bot such as OAI-SearchBot or PerplexityBot. Four block a training bot such as GPTBot or ClaudeBot, so the split between search and training is real but small. The robots.txt guide covers which bot does what.
• Firewalls are not the obstacle. Only 2 of 84 homepages answered an AI user agent differently from a regular request. For this group, at least, the edge is not quietly undoing what robots.txt allows. Our note on Cloudflare and AI crawlers explains why the edge matters elsewhere.
• The HTML carries the page. 6 of 83 homepages needed JavaScript for their main content. That is low, and it means the text an agent needs is usually in the first response. Do AI crawlers run JavaScript?
• llms.txt won; Markdown did not. 53 of 81 homepages publish an llms.txt, a format no AI search engine documents reading. Only 20 of 83 serve a Markdown version. The file is cheap; a second rendering of every page is not. llms.txt and Markdown for agents cover both.
• Structured data is the norm, not the exception. 56 of 84 homepages pass the core JSON-LD check. The remaining third is where an agent has to infer the company, the page or the date from prose. Structured data for AI search has the types worth adding.
• Agent protocols are still early. 11 of 84 homepages declare a Content-Signal, 12 publish an API or agent discovery document, and 1 exposes WebMCP tools. None publishes DNS-AID records. These are optional; the numbers say the formats have not reached mainstream adoption. The agent-readiness checklist sorts them by the kind of site that needs them.

Scores

The median homepage scored 77.6 out of 100 and the mean 74.8, from 30 to 89.7. Most cluster in the 70s and 80s: 62 of the 84 scored homepages sit between 70 and 89. No homepage reached an A (90 or more).

Where homepages lose points

These are the checks that fewest homepages passed. Weights and pass criteria are in the methodology.

Method and limits

• The sample is the 2025 edition of the Forbes Cloud 100, the latest published. Each company's homepage was scanned once from one machine in Europe with the standard CrawlWise checkset.
• 84 could be scored; 15 returned a block or challenge and 1 failed. Those sites are outside the check denominators, which is why every share says how many homepages it covers.
• A homepage is one page. A company can do better on its documentation, blog or product pages, and a homepage is not where an AI crawler always lands.
• AI user agents were sent from our address, not from the crawler operators' addresses. A firewall that verifies IPs can treat the real crawlers differently.
• Markdown counts when the homepage advertises it or serves it for Accept: text/markdown. Word overlap with the rendered page measures whether it matches; navigation and link URLs in converted Markdown lower that score.
• Performance figures come from one unthrottled browser visit. They are lab observations, not field Core Web Vitals or PageSpeed Insights scores.

Data

The per-check counts, with no site named: CSV · JSON.

The aggregate CSV and JSON are available under the Creative Commons Attribution 4.0 International license. The article text and the CrawlWise name are not covered by that license.

Cite this study: CrawlWise, “AI readiness of the Forbes Cloud 100” (2026), https://crawlwise.dev/research/cloud-100-ai-readiness-2026.