Talk to usGet started
ProductPricingMethodologyResourcesAgenciesTalk to usGet started

llms.txt, schema and robots.txt: the technical checklist for AI search

For AI search, the technical work that matters most is letting the right crawlers in and keeping your content readable as text. Allow each vendor’s search crawler, decide separately on training crawlers, check your firewall and CDN too, keep structured data accurate, and treat llms.txt as a cheap extra rather than a ranking lever.

Citino teamUpdated 5 min read

1. Let the right crawlers in

AI companies run different bots for different jobs. The most important distinction is search (being found and cited) versus training (being used to build future models). Each is controlled on its own.

VendorSearch crawlerUser-requested fetchTraining
OpenAIOAI-SearchBotChatGPT-UserGPTBot
PerplexityPerplexityBotPerplexity-UserNot crawled for training, per Perplexity
AnthropicClaude-SearchBotClaude-UserClaudeBot
GoogleGooglebot (Search, AI Overviews, AI Mode)n/aGoogle-Extended (also Gemini grounding)

Sources: OpenAI, Perplexity, Anthropic, Google.

Two things catch people out:

  • Google-Extended is not about Search. It does not affect AI Overviews, but it does control whether Gemini apps can ground answers in your content. Blocking it to avoid training also removes you from Gemini’s grounded answers.
  • User-requested fetchers may ignore robots.txt. OpenAI says its rules may not apply to ChatGPT-User; Perplexity says Perplexity-User generally ignores them, because a person asked for the page.

2. A sample robots.txt

Allow search and user fetches, and make an explicit choice on training. This example opts out of training at OpenAI and Anthropic but keeps Gemini grounding:

User-agent: Googlebot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Adjust the training lines to your own policy. Changes are not instant; OpenAI, for example, says robots.txt updates take about a day to apply.

3. Check the firewall and CDN, not just robots.txt

Many sites allow AI crawlers in robots.txt and then block them at the edge with a bot-protection rule. Check your CDN or firewall settings for AI-bot blocking, challenge pages and rate limits. If a crawler gets a challenge page instead of your content, robots.txt does not matter.

4. Keep content readable as text

  • Put important facts in the HTML, not only in images, video or downloadable PDFs.
  • Make sure content does not depend on client-side scripts to appear.
  • Use one H1, descriptive H2 and H3 headings, real tables for comparisons and real lists for steps.
  • Keep a sitemap, link related pages to each other, and show a visible “updated” date where facts change.

5. Structured data: accurate, not magic

Google’s guidance for AI features says there is “no special schema.org structured data that you need to add”, and that structured data should match the visible text. Structured data is still useful for search engines in general. The ones worth having for most B2B sites:

TypeUse it on
OrganizationYour home or about page: name, logo, URL, social profiles
Product or SoftwareApplicationProduct and pricing pages
ArticleGuides and blog posts, with dates
BreadcrumbListAny page with breadcrumbs
FAQPagePages with a real FAQ (Google shows FAQ rich results only for some sites, but the markup still describes the page)

6. llms.txt: a cheap extra

llms.txt is a proposal, published by Jeremy Howard in September 2024, for a Markdown file at your site’s root that summarises what the site is and links to its most useful pages. It is easy to add and some tools read it.

Keep expectations in proportion: Google says AI Overviews and AI Mode need no AI text files, and no major engine has said it uses llms.txt to choose sources. Add it if it is cheap for you; never let it stand in for the steps above. You can see Citino’s own llms.txt as an example.

The checklist

  1. Search crawlers allowed in robots.txt: Googlebot, OAI-SearchBot, PerplexityBot, Claude-SearchBot.
  2. Training crawlers set by explicit choice: GPTBot, ClaudeBot, Google-Extended.
  3. No firewall, CDN or bot-protection rule blocking the crawlers you allowed.
  4. Key facts in HTML text, not only in images, PDFs or scripts.
  5. Clear headings, real tables and lists, visible update dates.
  6. Accurate structured data that matches the page.
  7. Sitemap submitted and linked; related pages linked to each other.
  8. Optional: an llms.txt at the root.

Questions people ask

Should I block AI crawlers?

Decide by purpose. Blocking search crawlers such as OAI-SearchBot, PerplexityBot or Claude-SearchBot removes you from those engines’ search answers. Blocking training crawlers such as GPTBot or ClaudeBot keeps your content out of model training without affecting search. Many brands allow search and decide on training separately.

Does llms.txt help me rank in AI answers?

There is no public evidence that any major engine uses it as a ranking signal, and Google says AI Overviews need no AI text files. It is cheap to add and may help tools that read it, so it is reasonable to have, but it is not a substitute for crawlable, clear pages.

Does schema markup help AI Overviews?

Google says there is no special schema.org markup needed for AI Overviews or AI Mode, and that any structured data should match the visible text. Structured data still helps search engines understand your pages, so keep it accurate rather than adding it for AI.

Keep reading

See where AI names you
on every engine, every day

Citino asks the questions your buyers ask and tells you whether your fixes worked.

Get started Talk to us