# Guides How AI crawlers and agents read a site, and what to change so they can. - [AI crawlers in robots.txt: what to allow and block](https://crawlwise.dev/guides/ai-crawlers-robots-txt.md): GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and the rest: which AI bots train, which search, which fetch for users, and a robots.txt you can copy. - [Do AI crawlers run JavaScript? Mostly, no](https://crawlwise.dev/guides/ai-crawlers-javascript.md): GPTBot, ClaudeBot and PerplexityBot read the HTML your server sends, not what JavaScript renders. How to test a page and fix content that needs JavaScript. - [Is Cloudflare blocking AI crawlers on your site?](https://crawlwise.dev/guides/cloudflare-ai-crawlers.md): Cloudflare can block AI search, agent and training crawlers before they reach your server. Where the settings are, what each one does, and how to check. - [llms.txt: what it is and whether you need one](https://crawlwise.dev/guides/llms-txt.md): llms.txt is a Markdown index of your site for language models. What the proposal says, who actually reads it, and how to publish one in ten minutes. - [Serve Markdown to AI agents: .md pages and headers](https://crawlwise.dev/guides/markdown-for-agents.md): Give agents a clean Markdown version of each page: the URL convention, Accept: text/markdown, and the Link, Vary and canonical headers that keep it correct. - [Structured data for AI search: what helps and what doesn't](https://crawlwise.dev/guides/structured-data-ai-search.md): What Google and Microsoft say about schema.org markup in AI search, which types are worth adding, and a JSON-LD example you can adapt and test. - [Is your site agent-ready? A practical checklist](https://crawlwise.dev/guides/agent-ready-website-checklist.md): What makes a website agent-ready in 2026: access, readable content, structured data, Markdown and the protocols for APIs, sorted by what actually matters. - [Lighthouse Agentic Browsing: what it checks](https://crawlwise.dev/guides/lighthouse-agentic-browsing.md): The seven audits in Lighthouse's Agentic Browsing category, what each one needs to pass, and how to fix the accessibility tree, WebMCP and llms.txt findings. - [WebMCP: let AI agents use your site's forms](https://crawlwise.dev/guides/webmcp.md): WebMCP turns a form or a JavaScript function into a tool a browser agent can call. The declarative and imperative APIs, safety hints, and how to test them. --- Last updated 5 November 2026. HTML version: Every CrawlWise page in Markdown: