# Methodology CrawlWise uses a versioned checkset. The current version is **0.4.10**. Scores are reproducible: the same pages, checkset and scan region should produce the same result, apart from timing noise. A report keeps the checkset version it was scanned with, so updating the checkset never rewrites older reports. ## What a scan covers Each scan audits one to three pages on the same host: the URL you submit and up to two key pages you add. CrawlWise checks `robots.txt` for `CrawlWiseBot` before auditing any page, fetches the raw HTML and supporting files, then loads each page in Chromium at a 1365 × 900 desktop viewport. It does not sign in, submit forms, interact with controls, scroll the page or crawl the rest of the site. Site checks run once: robots.txt, AI bot access, sitemap, resource access, llms.txt, HTTPS and security headers. Page checks run for every audited page and are merged for the score: a check fails if it fails on any page, warns if it warns anywhere, and passes only if every page passes. Each page also gets a diagnostic page score, and the report lists page findings separately. The scanner identifies itself as `CrawlWiseBot/1.0` and runs from an EU region. A bot challenge or access denial is reported as **blocked**, not as a failed score. A consent wall is reported as a content finding instead. ## Scoring Each check has a weight. A pass earns the full weight, a warning half, a failure zero. Checks that do not apply are excluded from the denominator. If less than 80% of the weight can be evaluated, the scan is marked incomplete and no score is shown. Four critical failures cap the score: a noindex page caps at 30, missing HTTPS at 40, blocking all search and citation bots at 50, and serving no main content without JavaScript at 55. ## Check statuses | Status | Meaning | Scoring | | --- | --- | --- | | Pass | The page meets the criterion. | Full weight | | Warning | The page partially meets it or is close to a boundary. | Half weight | | Fail | The page misses the criterion. | Zero | | N/A | The check does not apply, for example no dated structured data. | Excluded | | Blocked | The target or a specific check was blocked. | Excluded | | Error | The check could not run. | Excluded | A fully blocked target has no score. Individual blocked or errored checks are excluded from the denominator; too many of them leave too little weight to evaluate and mark the scan incomplete. ## Grades | Grade | Score | What it means | | --- | --- | --- | | **A** | 90–100 | Excellent agent readiness. The page is easy to fetch, read and trust. | | **B** | 80–89 | Strong result. A small number of fixes would remove the remaining friction. | | **C** | 70–79 | Fair. Several checks need attention before the page is reliably agent-ready. | | **D** | 60–69 | Weak. Important signals are missing or broken. | | **E** | 50–59 | Poor. Most checks need work. | | **F** | 0–49 | Critical. The page is likely hard or impossible for agents to use. | ## Categories | Category | What it covers | Weight | | --- | --- | --- | | Discovery | Can crawlers and AI agents find the page? Covers robots.txt, citation bot access and sitemap discovery. | 27 | | Content | What a visitor or agent gets without JavaScript: main text, page structure, boilerplate and consent walls. | 20 | | Schema | Structured data: JSON-LD parses, core fields are present and dates are valid. | 15 | | Agent | Agent-specific signals: a Markdown version of the page and an llms.txt file. | 12 | | Performance | Fetch and render performance: TTFB, HTML payload, caching, resource budget and lab LCP/CLS. | 15 | | Trust | Security and trust: HTTPS, HSTS, security headers and redirect behaviour. | 8 | | Freshness & A11y | Machine-readable dates and automated accessibility checks. | 3 | | API & agent protocols | Discovery documents for sites with an API or app: API catalog, OAuth metadata, MCP and A2A cards, agent skills and WebMCP tools. Not scored. | 0 | ## Checks ### Discovery (27) Can crawlers and AI agents find the page? Covers robots.txt, citation bot access and sitemap discovery. - `discovery.robots` — robots.txt is reachable and valid (weight 2) - `discovery.ai-bots` — AI search and citation bots are allowed (weight 6) - `discovery.ai-bot-access` — AI crawlers get the same page as a regular fetch (weight 2) - `discovery.content-signal` — Content usage preferences for AI are declared (weight 0) - `discovery.sitemap` — sitemap.xml is valid and current (weight 2) - `discovery.indexability` — The page is indexable and has a consistent canonical (weight 8) - `discovery.redirects` — Redirects are permanent and short (weight 3) - `discovery.resource-access` — robots.txt allows CSS and JavaScript needed for rendering (weight 3) - `discovery.hreflang` — hreflang annotations are complete (weight 1) ### Content (20) What a visitor or agent gets without JavaScript: main text, page structure, boilerplate and consent walls. - `content.no-js` — Main content is available without JavaScript (weight 7) - `content.structure` — Document structure is extractable (weight 5) - `content.boilerplate` — Main content is not buried in boilerplate (weight 1) - `content.consent-wall` — Content is not hidden behind a consent wall (weight 1) - `content.images` — Content images are described and efficient (weight 3) - `content.links` — Internal links are crawlable and descriptive (weight 2) - `content.social` — Social preview metadata is complete (weight 1) ### Schema (15) Structured data: JSON-LD parses, core fields are present and dates are valid. - `schema.parses-core` — JSON-LD parses and describes the page (weight 8) - `schema.dates` — Structured data dates are valid (weight 3) - `schema.entities` — Structured data describes the page entity (weight 4) ### Agent (12) Agent-specific signals: a Markdown version of the page and an llms.txt file. - `agent.markdown` — Markdown is available for agents (weight 8) - `agent.llms-txt` — llms.txt directory is present (weight 2) - `agent.markdown-negotiation` — Markdown is advertised and negotiated correctly (weight 2) ### Performance (15) Fetch and render performance: TTFB, HTML payload, caching, resource budget and lab LCP/CLS. - `perf.ttfb` — Time to first byte is reasonable (weight 2) - `perf.payload` — HTML payload is small and compressed (weight 2) - `perf.caching` — HTML has validators and cache headers (weight 2) - `perf.resources` — Page resources stay within a reasonable budget (weight 3) - `perf.lcp` — Largest Contentful Paint is under 2.5 seconds (weight 3) - `perf.cls` — Cumulative Layout Shift is under 0.1 (weight 3) ### Trust (8) Security and trust: HTTPS, HSTS, security headers and redirect behaviour. - `trust.https` — HTTPS, HSTS and redirects are correct (weight 6) - `trust.headers` — Security headers are present (weight 2) ### Freshness & A11y (3) Machine-readable dates and automated accessibility checks. - `freshness.signals` — Freshness signals are present (weight 1) - `a11y.axe` — Automated accessibility checks pass (weight 2) ### API & agent protocols (0) Discovery documents for sites with an API or app: API catalog, OAuth metadata, MCP and A2A cards, agent skills and WebMCP tools. Not scored. - `api.webmcp` — WebMCP tools are complete (weight 0) - `api.catalog` — An API catalog lists the site's APIs (weight 0) - `api.oauth-discovery` — OAuth or OpenID Connect discovery is published (weight 0) - `api.oauth-protected-resource` — OAuth protected resource metadata is published (weight 0) - `api.mcp-server-card` — An MCP server card describes the site's MCP server (weight 0) - `api.a2a-agent-card` — An A2A agent card describes the site's agent (weight 0) - `api.agent-skills` — An agent skills index lists the site's skills (weight 0) - `api.ard` — An ARD manifest lists the site's agent resources (weight 0) --- Last updated 4 November 2026. HTML version: Every CrawlWise page in Markdown: