Technical GEO for marketing engineers: robots.txt, structured data, llms.txt

What engineering owns in GEO

Content strategy may sit with marketing, but access and rendering are infrastructure concerns. If the relevant retrieval path cannot fetch or parse a page, that path cannot use it. Structured data remains useful for rich results and machine-readable facts, but generic schema has not shown a reliable citation lift. llms.txt is an optional machine-readable layer, not a ranking signal.

1. Admit the AI crawlers

Search, user-initiated and training bots

Not all bots do the same job, and vendor documentation is the source of truth.

• Search and indexing: OAI-SearchBot controls ChatGPT Search inclusion; Claude-SearchBot builds Anthropic’s search index; PerplexityBot supports Perplexity retrieval; Googlebot powers Google Search, including AI Overviews and AI Mode.
• User-initiated retrieval: ChatGPT-User and Claude-User visit pages in response to a user action. OpenAI explicitly says ChatGPT-User is not the control for Search inclusion.
• Training and model development: GPTBot and ClaudeBot collect public-web material that may contribute to model training. They are not the corresponding vendors’ search-inclusion controls.

OtterlyAI reported crawler barriers on 73% of sites in a dataset of more than one million citations. It is a vendor snapshot, so use it as a reason to inspect your own robots.txt, edge rules and logs, not as a universal base rate.

# Search/indexing — allow the surfaces you want to be eligible for
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

# User-initiated retrieval — separate from automatic search inclusion
User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

# Model development — an independent policy decision
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

Treat that as an illustrative policy, not a block to paste unchanged. Decide per vendor, keep private paths protected, and re-check official documentation. User agents can be spoofed, so validate actual traffic against published IP ranges where the vendor provides them.

Also check that your CDN or WAF bot protection isn’t silently blocking AI crawlers. An allow rule in robots.txt means nothing if the firewall returns 403.

For Google, Googlebot is the Search control. Google-Extended controls whether already crawled content can be used for training future Gemini models and for grounding in certain non-Search Gemini products; Google says it does not affect Search inclusion or ranking. Google also exposes generative-AI participation controls in Search Console for eligible properties, so check the current Search Console state rather than treating Google-Extended as an AI Overviews switch.

HTML that doesn’t depend on JavaScript

Vercel’s network study found no JavaScript execution in 569 million GPTBot requests. Googlebot can render crawlable JavaScript. SSR, SSG or prerendering remains the safest cross-platform baseline because critical body text is present in the initial HTML without depending on a particular fetcher’s rendering stack.

Verification is one command:

curl -A "OAI-SearchBot" https://example.com/page/ | grep "key copy"

If the copy isn’t in the initial HTML, the rendering strategy needs work.

2. Structured data: publish facts, not an AI-citation promise

Two facts come first. Google states that structured data is not required for its generative Search features and that there is no special schema.org markup to add for AI. Ahrefs tracked 1,885 pages that added JSON-LD against matched controls and found no meaningful citation uplift across AI Overviews, AI Mode or ChatGPT.

One observational result is worth testing in the right context, but it is not a proven universal exception:

• Product / SoftwareApplication + Offer with real attributes — a cross-platform observational study found that pages with attribute-rich Product/Review schema were cited in 61.7% of its sample versus 41.6% for generic schema types. The association was stronger for lower-authority domains, but the design does not establish that adding schema caused the difference. Explicit price also appeared as a citation gatekeeper in a separate controlled study. The useful principle is to publish accurate visible facts; do not promise a citation lift from the tag itself.
• Organization — use consistent legal name, address and official-profile sameAs links as machine-readable entity facts. Do not treat the markup itself as a citation signal.
• FAQPage — the wrapper alone does not establish a citation effect. Use it only where the page genuinely contains questions and substantive answers with the evidence they need.
• Article + Person — accurate authorship and dates are useful page metadata. They do not create an AI citation entitlement.

After implementation, validate with both the Rich Results Test and the Schema.org validator.

3. llms.txt: an optional supporting file

Honest framing first: llms.txt is not a demonstrated ranking or citation factor. Google says its generative Search features do not require AI text files, and industry reviews have found no credible citation effect. Some OpenAI traffic has been observed requesting llms.txt; crawling a file is not evidence that it changes answers.

When it can be generated from the site’s existing content source, a Markdown summary at the site root can be maintained with low overhead. Treat it as preparation for clients that may adopt it, below access, indexability and visible content in priority.

Operationally, a hand-maintained llms.txt can drift as pages change. Generate it from the same content source at build time where practical; this site’s file is generated from its content collections.

Verification checklist

1. robots.txt reflects the correct bot purpose: search/indexing, user-initiated retrieval or training
2. WAF/CDN doesn’t 403 AI crawler user agents (check access logs)
3. a direct fetch shows critical body copy in the initial HTML
4. where relevant, product pages expose accurate visible price/specification facts and matching Product/Offer data
5. if llms.txt is published, it is generated from the content source (optional, not a citation factor)
6. Server logs show real AI crawler visits, validated against official IP ranges and reviewed regularly

This foundation removes known access and extraction obstacles. Repeated measurement can then show whether citations or mentions changed after publication, but that observation alone does not establish causality.

The source list above includes the current vendor documentation and the studies discussed. Re-check bot names and controls before changing production crawler policy; these interfaces can change.