# CrawlWise > CrawlWise is a technical audit tool that shows website owners how a public page appears to crawlers and AI agents. CrawlWise fetches a public page and up to two same-host key pages, compares their raw HTML with a browser rendered version, and produces a private report with evidence and prioritized technical fixes. It evaluates access, content visibility, metadata, structured data, and agent friendly formats. It does not measure brand mentions or guarantee inclusion in AI answers. - Only public HTTP and HTTPS URLs on standard ports are accepted. - The scanner does not bypass authentication, paywalls, or bot protection. - The scanner honors robots.txt rules that disallow CrawlWiseBot. - Reports are private links and expire after 30 days. - A report contains a score when enough checks can be evaluated, category breakdowns, findings, and evidence. - A high score is not a promise of search ranking or AI citations. General questions, abuse and security: contact@crawlwise.dev ## Docs - [Home](https://crawlwise.dev/index.md): what the audit covers and how to start a free scan - [Methodology](https://crawlwise.dev/methodology.md): every check with its weight, pass/warn/fail criteria, grade bands and score caps - [Sample report](https://crawlwise.dev/sample-report.md): a real CrawlWise report of imagineit.dev with its score, findings, evidence and fixes - [AI readiness of the Forbes Cloud 100](https://crawlwise.dev/research/cloud-100-ai-readiness-2026.md): how the 100 companies of the Forbes Cloud 100 read to AI crawlers and agents, with the aggregate data - [AI readiness of the top 100 e-commerce sites](https://crawlwise.dev/research/ecommerce-100-ai-readiness-2026.md): how the 100 most-visited e-commerce websites read to AI crawlers and agents, with the aggregate data - [Report abuse](https://crawlwise.dev/abuse.md): report removal, opting a site out of scans and security disclosure - [Privacy](https://crawlwise.dev/privacy.md): what a scan stores, for how long, and how to have a report removed - [Free tools](https://crawlwise.dev/tools.md): free checkers for AI crawler access, robots.txt, sitemaps, redirects, indexability, llms.txt, structured data, Open Graph, security headers and agent discovery - [AI crawler checker](https://crawlwise.dev/tools/ai-crawler-checker.md): Free AI crawler checker: test if GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot can reach your site through robots.txt, Cloudflare and your firewall. - [llms.txt checker](https://crawlwise.dev/tools/llms-txt-checker.md): Free llms.txt checker: see if your file is reachable, is Markdown, follows the llmstxt.org layout, and whether your pages serve Markdown to AI agents. - [Can AI read your site without JavaScript?](https://crawlwise.dev/tools/ai-javascript-checker.md): Free test of what AI crawlers see: compare your raw HTML with the rendered page to find content that ChatGPT, Claude and Perplexity miss without JavaScript. - [Structured data checker for AI search](https://crawlwise.dev/tools/structured-data-checker.md): Free JSON-LD checker for AI search: see if your schema.org markup parses, is in the server HTML, describes the page entity and carries valid dates. - [WebMCP and agent discovery checker](https://crawlwise.dev/tools/mcp-agent-discovery-checker.md): Free checker for agent protocols: WebMCP tools, MCP server cards, A2A agent cards, agent skills, API catalogs and OAuth discovery documents on your site. - [robots.txt checker](https://crawlwise.dev/tools/robots-txt-checker.md): Free robots.txt checker: see if your file is reachable and valid, which search and AI bots it allows, and whether Googlebot can fetch your CSS and JavaScript. - [XML sitemap checker](https://crawlwise.dev/tools/sitemap-checker.md): Free XML sitemap checker: see if your sitemap.xml is reachable and valid, is declared in robots.txt, lists the scanned page and carries lastmod dates. - [Redirect checker](https://crawlwise.dev/tools/redirect-checker.md): Free redirect checker: trace every hop from the URL you enter to the final page, spot temporary redirects and long chains, and check HTTPS and HSTS. - [Indexability and canonical checker](https://crawlwise.dev/tools/indexability-checker.md): Free indexability checker: find noindex, nofollow and nosnippet in meta tags and X-Robots-Tag headers, check the canonical URL and validate hreflang. - [Open Graph and meta tag checker](https://crawlwise.dev/tools/open-graph-checker.md): Free Open Graph checker: see if your page has og:title, og:description and an absolute og:image, plus a title, meta description, H1 and lang attribute. - [Security headers checker](https://crawlwise.dev/tools/security-headers-checker.md): Free security headers checker: test Content-Security-Policy, X-Content-Type-Options, Referrer-Policy, frame protection and HSTS on your page. - [Guides](https://crawlwise.dev/guides.md): practical guides to making a site readable for AI crawlers and agents - [AI crawlers in robots.txt: what to allow and block](https://crawlwise.dev/guides/ai-crawlers-robots-txt.md): GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and the rest: which AI bots train, which search, which fetch for users, and a robots.txt you can copy. - [Do AI crawlers run JavaScript? Mostly, no](https://crawlwise.dev/guides/ai-crawlers-javascript.md): GPTBot, ClaudeBot and PerplexityBot read the HTML your server sends, not what JavaScript renders. How to test a page and fix content that needs JavaScript. - [Is Cloudflare blocking AI crawlers on your site?](https://crawlwise.dev/guides/cloudflare-ai-crawlers.md): Cloudflare can block AI search, agent and training crawlers before they reach your server. Where the settings are, what each one does, and how to check. - [llms.txt: what it is and whether you need one](https://crawlwise.dev/guides/llms-txt.md): llms.txt is a Markdown index of your site for language models. What the proposal says, who actually reads it, and how to publish one in ten minutes. - [Serve Markdown to AI agents: .md pages and headers](https://crawlwise.dev/guides/markdown-for-agents.md): Give agents a clean Markdown version of each page: the URL convention, Accept: text/markdown, and the Link, Vary and canonical headers that keep it correct. - [Structured data for AI search: what helps and what doesn't](https://crawlwise.dev/guides/structured-data-ai-search.md): What Google and Microsoft say about schema.org markup in AI search, which types are worth adding, and a JSON-LD example you can adapt and test. - [Is your site agent-ready? A practical checklist](https://crawlwise.dev/guides/agent-ready-website-checklist.md): What makes a website agent-ready in 2026: access, readable content, structured data, Markdown and the protocols for APIs, sorted by what actually matters. - [Lighthouse Agentic Browsing: what it checks](https://crawlwise.dev/guides/lighthouse-agentic-browsing.md): The seven audits in Lighthouse's Agentic Browsing category, what each one needs to pass, and how to fix the accessibility tree, WebMCP and llms.txt findings. - [WebMCP: let AI agents use your site's forms](https://crawlwise.dev/guides/webmcp.md): WebMCP turns a form or a JavaScript function into a tool a browser agent can call. The declarative and imperative APIs, safety hints, and how to test them. ## Optional - [Full text](https://crawlwise.dev/llms-full.txt): the Markdown of every page above in one file