Methodology

What a scan covers

Each scan audits one to three pages on the same host: the URL you submit and up to two key pages you add. CrawlWise checks robots.txt for CrawlWiseBot before auditing any page, fetches the raw HTML and supporting files, then loads each page in Chromium at a 1365 × 900 desktop viewport. It does not sign in, submit forms, interact with controls, scroll the page or crawl the rest of the site.

Site checks run once: robots.txt, AI bot access, sitemap, resource access, llms.txt, HTTPS and security headers. Page checks run for every audited page and are merged for the score: a check fails if it fails on any page, warns if it warns anywhere, and passes only if every page passes. Each page also gets a diagnostic page score, and the report lists page findings separately.

The scanner identifies itself as CrawlWiseBot/1.0 and runs from an EU region. A bot challenge or access denial is reported as blocked, not as a failed score. A consent wall is reported as a content finding instead.

Scoring

Each check has a weight. A pass earns the full weight, a warning half, a failure zero. Checks that do not apply are excluded from the denominator. If less than 80% of the weight can be evaluated, the scan is marked incomplete and no score is shown.

Four critical failures cap the score: a noindex page caps at 30, missing HTTPS at 40, blocking all search and citation bots at 50, and serving no main content without JavaScript at 55.

Check statuses

A fully blocked target has no score. Individual blocked or errored checks are excluded from the denominator; too many of them leave too little weight to evaluate and mark the scan incomplete.

Grades

The score maps to a letter grade so reports are easy to compare at a glance.

Categories

Performance measurements

Performance checks include Time to First Byte, HTML size and caching, plus laboratory Largest Contentful Paint (LCP) and Cumulative Layout Shift (CLS) measurements from Chromium. The scan also records the page's network requests, total transfer, largest resources and render-blocking scripts. It names the element behind LCP and the top layout-shift sources. The scan uses an unthrottled browser and observes the initial viewport without simulated user interaction. These measurements help identify regressions; they are not Google PageSpeed Insights scores or field Core Web Vitals.

LCP passes at 2.5 seconds or less and CLS at 0.1 or less. Metrics are captured before the evidence screenshots, so a full-page capture cannot turn lazy below-the-fold images into a late LCP. Network, device and page state can change results between scans. Interaction to Next Paint (INP) is not reported because a single automated visit cannot represent real user interactions.

The thresholds follow published Core Web Vitals boundaries. Google's field classification uses the 75th percentile across visits, so this single-scan result is a diagnostic signal, not a field classification.

Page resources. The resource check passes at or below 1 MB transfer and 50 requests with no render-blocking scripts. It warns up to 2 MB, 80 requests and two render-blocking scripts, and fails above those limits. Sizes include response headers; a render-blocking script is a <script src> without async or defer. Module scripts (type="module") are deferred by default and are not counted as render-blocking.

Evidence and screenshots

Every check keeps machine-readable evidence. The report shows labelled values and thresholds, with a short explanation for performance metrics, and keeps the raw JSON available. Reports include an initial desktop viewport screenshot, a full-page desktop capture and a mobile viewport screenshot. Screenshots are private evidence and expire after seven days. The "What an agent sees" section shows the text extracted from HTML and, when the site serves it, the Markdown representation that agents can request or discover through a link. Reports from key-page audits show a score and findings for each audited page alongside the merged scan result.

Checks in 0.4.10

Discovery

Content

Schema

Agent

API & agent protocols

For sites with an API, an app or actions agents can take. These checks carry no weight, appear in their own report tab, and a content site can ignore them.

Performance

Trust

Freshness & A11y

What this is not

CrawlWise is not a ranking tool and does not measure brand mentions in AI answers. It is a technical audit of what a crawler or agent can fetch, parse and quote. A high score does not guarantee citations; a low score is a concrete list of things to fix.

llms.txt is marked experimental because it is a community proposal, not a standard, and absence of the file does not block search or citation.