# Indexability and canonical checker

Check whether search engines and AI search may index the page: robots meta tags, the X-Robots-Tag header, the canonical URL and, for multilingual pages, the hreflang set.

Run it at https://crawlwise.dev/tools/indexability-checker: paste a public URL, no account needed. The scan runs the full CrawlWise audit and returns a private report.

## What this checker tests

- The page is indexable and has a consistent canonical (`discovery.indexability`)
- hreflang annotations are complete (`discovery.hreflang`)

## What the report shows

- robots meta tags (`robots`, `googlebot`, `bingbot`) and the `X-Robots-Tag` response header;
- whether any of them says `noindex` or `none`, which fails, or `nofollow`, `nosnippet` or
  `max-snippet:0`, which warns. A page without snippets can be indexed but gives AI answers
  nothing to quote;
- the canonical URL, whether it is the page's own final URL, and how many canonical tags the
  page has;
- for pages with hreflang annotations, whether the set includes the page itself and
  `x-default`, and whether any language appears twice.

## Common mistakes

- **A `noindex` left over from staging**, often in an `X-Robots-Tag` header set by the server or
  CDN rather than in the HTML.
- **A canonical to the home page** on every page, copied from a layout.
- **A canonical to the `http://` or non-`www` version**, which the final URL redirects away from.
- **Canonical tags added by client-side JavaScript**, which crawlers that read raw HTML never see.

## Fix it

Remove `noindex` from pages you want found, and give every page one canonical tag in the server
HTML that points to its own final URL. For multilingual sites, list every language version on
every page, including the page itself and an `x-default`. The
[JavaScript visibility test](/tools/ai-javascript-checker) shows what else crawlers miss in the
raw HTML.

## FAQ

### Why is my page not indexed?

The most common technical reasons are a noindex directive in a robots meta tag or an X-Robots-Tag header, a canonical tag that points to another URL, or a robots.txt rule that blocks the page. This checker covers the first two; the robots.txt checker covers the third.

### Does CrawlWise read the canonical from the rendered page?

No. It reads the server HTML, the way most crawlers first see the page. A canonical or noindex added by JavaScript after load is missed by crawlers that do not render, so put both in the server response.

### What is a self-referencing canonical?

A canonical tag that points to the page's own final URL. It tells crawlers which version to index when the same content is reachable with parameters, a trailing slash or another host. A missing, foreign or duplicated canonical warns.

### When does the hreflang check apply?

Only when the page has hreflang annotations; otherwise it is not applicable and does not change the score. A complete set includes the page itself and an x-default entry. Two entries for the same language fail.

## Related guides

- [Do AI crawlers run JavaScript? Mostly, no](https://crawlwise.dev/guides/ai-crawlers-javascript.md)

---

Last updated 19 November 2026.

HTML version: <https://crawlwise.dev/tools/indexability-checker>

Every CrawlWise page in Markdown: <https://crawlwise.dev/llms.txt>
