Glossary

AI crawler

Bots AI companies use to fetch the web, such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. Direct AI crawlers often skip JavaScript; Googlebot can render crawlable JS.

Notes

One distinction matters most: training crawlers (e.g. GPTBot, collecting model training data) and retrieval bots (OAI-SearchBot, ChatGPT-User — fetching in real time when generating answers) are separate and can be controlled independently in robots.txt. If you want citations, retrieval bots must be allowed.

Second key fact: Vercel found no JavaScript execution in GPTBot requests. Googlebot can render crawlable JavaScript for Search, including AI Overviews. Critical copy in the initial HTML is the safest cross-platform baseline. CDNs/WAFs sometimes block AI bots by default — check CDN settings, not just robots.txt.

Updated: Jun 10, 2026