Do AI crawlers run JavaScript? Mostly, no

Google has rendered JavaScript for years, so many teams assume every crawler does. AI crawlers mostly do not. If your page text appears only after JavaScript runs, ChatGPT, Claude and Perplexity are likely to see an empty shell where your content should be.

What the crawlers actually do

In December 2024, Vercel and MERJ published an analysis of AI crawler traffic across Vercel’s network. Their conclusion: “none of the major AI crawlers currently render JavaScript.” The crawlers they checked include:

• OpenAI: GPTBot, OAI-SearchBot and ChatGPT-User
• Anthropic: ClaudeBot
• Perplexity: PerplexityBot
• Meta: Meta-ExternalAgent
• ByteDance: Bytespider
• Common Crawl: CCBot

These crawlers do download JavaScript files: 11.5% of ChatGPT’s requests and 23.8% of Claude’s were for JavaScript. They just do not execute them.

Two crawlers render pages like a browser:

• Googlebot. Gemini uses Googlebot’s infrastructure, so Google’s AI features see rendered pages.
• Applebot. Apple says it may render pages in a browser, and the Vercel study observed it doing so.

The study was a snapshot, and crawlers change. The safe assumption is still the same: content that is not in the HTML response may not exist for an AI system.

What disappears without JavaScript

Anything your server does not send in the HTML response:

• Single-page app shells. A page that arrives as <div id="root"></div> plus a bundle has no content for a crawler that does not run the bundle.
• Content fetched after load. Text, prices or reviews that a component loads from an API after it mounts.
• Metadata added by scripts. A title, description, canonical or JSON-LD block that a client-side library writes into the head.
• Content behind interaction. “Load more” buttons, infinite scroll and tabs whose content is requested only when clicked. Tabs and accordions are fine if their content is already in the HTML.
• Content behind a consent or login wall that is lifted by a script.

Vercel notes one partial exception: data included in the initial HTML, such as JSON or React Server Components payloads, “may still be indexed.” That is data, not readable text, so do not rely on it for content you want cited.

Test a page in a minute

Pick a sentence that only appears in your page’s main content, then fetch the raw HTML:

curl -sS https://example.com/pricing | grep -c "Billed annually"

0 means the sentence is not in the HTML your server sends. You can do the same with a crawler’s user agent, which also shows whether a firewall treats bots differently:

curl -sS -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://example.com/pricing | grep -c "Billed annually"

In a browser, open DevTools, run Disable JavaScript from the command menu and reload. What you still see is roughly what an AI crawler gets. View page source shows the raw HTML, while the Elements panel shows the page after scripts have run.

A CrawlWise scan does this for every audited page: it compares the text in the raw HTML with the text after rendering in Chromium, and reports how much of the content needs JavaScript.

How to fix it

The goal is simple: the HTML response contains the page’s main content. Interactive parts can still use JavaScript.

Next.js

In the App Router, Server Components are the default, and they render on the server. Content goes missing when it is fetched in a Client Component inside useEffect. Fetch it in a Server Component instead, or generate the page at build time with static generation. Keep "use client" for the interactive parts, not for whole pages.

Nuxt, SvelteKit, Remix and Astro

These frameworks render on the server by default. Check for pages or routes where server rendering has been switched off, such as ssr: false in Nuxt, and for content loaded only in client-side hooks.

React, Vue or Angular single-page apps

A client-rendered app built with Vite or a similar tool has no server-rendered content. Move the public pages to a framework that renders on the server, use Angular’s server-side rendering, or prerender the public routes to static HTML at build time. Marketing pages, documentation and product pages matter most; a logged-in dashboard does not need to be crawlable.

Rails, Django, Laravel and other server-rendered apps

Server views already send the content. Watch for parts added later with JavaScript, for example a Turbo Frame with a src that loads after the page, or a widget that fetches its content from an API in the browser. If that part matters, render it in the initial response.

WordPress and site builders

Most themes render on the server. Third-party widgets are the usual gap: reviews, pricing tables or FAQs injected by an external script. Prefer blocks or plugins that output HTML.

Do not serve bots a different page

Serving prerendered HTML only to bots, known as dynamic rendering, is a workaround that Google does not recommend as a long-term solution. It doubles what you have to keep correct, and if the bot version differs from what people see, it looks like cloaking. Render the same HTML for everyone.

Sources

• Vercel and MERJ: The rise of the AI crawler
• Apple: About Applebot
• Google: Dynamic rendering as a workaround
• Google: Understand the JavaScript SEO basics