Skip to content
Log inTest my visibilityStart free trial
Log in
Language English
Start free trial Test my visibility

Let AI crawlers read your pages under robots.txt

Technical Updated Oct 4, 2026

Description

This criterion reads your site’s robots.txt file the way AI crawlers do, following the RFC 9309 standard. It checks that the file responds without error and that AI search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) can read the page, as well as Google-Extended, the token that controls how Gemini uses your pages; then the crawlers that fetch it on behalf of a user, the training crawlers and your llms.txt file.

A single blocked search crawler makes the criterion bad. Crawlers that ignore robots.txt according to their publisher, such as ChatGPT-User, are not counted.

Why it matters for AI engines

An assistant only cites pages its crawler was able to read. A Disallow rule, a robots.txt in error or an overly strict firewall can be enough to remove your page from the answers of ChatGPT, Perplexity or Claude. Google also recommends making sure crawling is allowed by robots.txt and by your CDN or hosting provider.

  • Silence means permission: a crawler with no group of its own follows the User-agent: * group, and a page that no rule disallows is open to it. There is no need to allow each crawler by name.
  • A robots.txt that answers with a 5xx or 429 error, confirmed by a second try, makes crawlers that follow RFC 9309 treat the whole site as disallowed. A missing file (404) disallows nothing.
  • Blocking training crawlers (GPTBot, ClaudeBot, CCBot…) keeps your pages out of future training data without shutting out search crawlers: it is a legitimate choice, to be made knowingly.

How to fix it

  1. 1 Make robots.txt reachable and readable The /robots.txt address must answer 200 with a text file, or 404 if the file does not exist. Fix any 5xx or 429 error, or an HTML page served in its place. A firewall or CDN can block AI crawlers without touching robots.txt: check its settings. Faced with a challenge page, iPand cannot conclude.
  2. 2 Open the page to search and user crawlers In the User-agent: OAI-SearchBot, PerplexityBot, Claude-SearchBot, Google-Extended, Claude-User or MistralAI-User groups, and in the * group, remove the Disallow rules that cover the page. A crawler with no group of its own follows the * group: not disallowing it is enough. OpenAI says a robots.txt change takes about 24 hours to apply.
  3. 3 Decide about training crawlers GPTBot, ClaudeBot, CCBot, Applebot-Extended and Meta-ExternalAgent collect pages to train models. Allowing them lets your texts into that data; blocking them asks that your texts no longer be used, with no effect on search crawlers. Google-Extended is the exception: it is not a crawler but a token, read by Google’s crawlers, that controls both training and the grounding of Gemini’s answers. Blocking it does not remove your site from Google Search or from AI Overviews, which depend on Googlebot.
  4. 4 Fix or remove llms.txt llms.txt is a proposal (llmstxt.org), not a standard: a Markdown file at the root of the site that starts with a # heading naming the site, followed by links to your key pages. If it exists, it must answer 200 as text, not HTML. Its absence (404 or 410) is not counted, and this check carries little weight.
  5. 5 Let iPandBot analyze the page iPand reads your pages with its own crawler, iPandBot (user agent iPandBot/1.0 (+https://ipand.io)). It respects a Disallow rule placed in a group that names it, User-agent: iPandBot, and that covers the page: the analysis then stops and the page is not analyzed. A User-agent: * group does not stop it, because the site owner is the one requesting the analysis.

Key takeaway

A blocked search crawler is an AI that cannot cite your page. Only block what you have chosen to block, and keep a robots.txt that answers without errors.

Further reading