What AI crawler rendering data says about your initial HTML

What we actually know

Vercel, together with MERJ, analyzed crawler traffic across its network in late 2024. It observed 569 million GPTBot requests in one month and found no JavaScript execution. GPTBot fetched JavaScript files in 11.5% of its requests, but the study found no evidence that it ran them. That is an empirical GPTBot snapshot, not a permanent capability guarantee for every AI product.

OpenAI documents three separate roles. OAI-SearchBot controls inclusion in ChatGPT Search, ChatGPT-User makes user-initiated visits and is not used to determine Search inclusion, and GPTBot collects data that may be used to train foundation models. Anthropic likewise separates Claude-SearchBot for search, Claude-User for user-directed retrieval and ClaudeBot for model development. Audit the bot that matches the surface you care about.

A client-rendered React or Vue page can return little more than an application shell to a non-rendering fetcher. If product facts and prices appear only after hydration, those fetchers may miss them. Google is the important counterexample: Googlebot can render crawlable JavaScript and supplies Google Search, including AI Overviews.

Check your site in three minutes

1. Disable JavaScript in your browser’s dev tools and reload key pages (product, pricing, company). If the content remains visible, the critical copy does not depend on client-side JavaScript.
2. Or right-click → “View page source” and confirm the body text exists in the source HTML. A bare <div id="root"></div> is the warning sign.
3. Check CDN and WAF behavior as well. A permissive robots.txt does not help if the edge returns 403 or 429. OtterlyAI reported technical crawler barriers on 73% of sites in its one-million-citation dataset; this is a vendor study, so treat the percentage as a snapshot and verify your own logs.

Your options

• Static generation (SSG) — first choice for content sites: HTML is complete at build time, which wins on crawlers, speed and operations. This site works this way.
• Server-side rendering (SSR) — for pages that need real-time data; the server returns HTML per request.
• Isolate interactivity — instead of a full SPA, run only the widgets that need it on the client (islands architecture) so the main body can remain in the initial HTML.

Googlebot can render JavaScript, but a page still has to be crawlable, indexed and eligible for a Search snippet to appear as a supporting link in AI Overviews or AI Mode. Google says there are no extra technical requirements for those features. Initial HTML remains the safest cross-platform baseline because it does not depend on each fetcher’s rendering capability.

Summary

Most GEO advice focuses on what to write. Before that, verify the retrieval paths relevant to the target platforms: bot purpose, robots controls, CDN behavior, HTTP response and initial HTML. Passing those checks establishes readiness. It does not predict that a platform will retrieve or cite the page.

Related terms: AI crawler, RAG, GEO