Guides
Guides
How AI crawlers and agents read a site, and what to change so they can. Each guide links to the CrawlWise checks that test it.
-
AI crawlers in robots.txt: what to allow and block
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and the rest: which AI bots train, which search, which fetch for users, and a robots.txt you can copy.
-
Do AI crawlers run JavaScript? Mostly, no
GPTBot, ClaudeBot and PerplexityBot read the HTML your server sends, not what JavaScript renders. How to test a page and fix content that needs JavaScript.
-
Is Cloudflare blocking AI crawlers on your site?
Cloudflare can block AI search, agent and training crawlers before they reach your server. Where the settings are, what each one does, and how to check.
-
llms.txt: what it is and whether you need one
llms.txt is a Markdown index of your site for language models. What the proposal says, who actually reads it, and how to publish one in ten minutes.
-
Serve Markdown to AI agents: .md pages and headers
Give agents a clean Markdown version of each page: the URL convention, Accept: text/markdown, and the Link, Vary and canonical headers that keep it correct.
-
Structured data for AI search: what helps and what doesn't
What Google and Microsoft say about schema.org markup in AI search, which types are worth adding, and a JSON-LD example you can adapt and test.
-
Is your site agent-ready? A practical checklist
What makes a website agent-ready in 2026: access, readable content, structured data, Markdown and the protocols for APIs, sorted by what actually matters.
-
Lighthouse Agentic Browsing: what it checks
The seven audits in Lighthouse's Agentic Browsing category, what each one needs to pass, and how to fix the accessibility tree, WebMCP and llms.txt findings.
-
WebMCP: let AI agents use your site's forms
WebMCP turns a form or a JavaScript function into a tool a browser agent can call. The declarative and imperative APIs, safety hints, and how to test them.
Last updated .