robots.txt and AI crawlers: the setting to check

Does blocking an AI crawler remove you from AI answers?

Yes, if you block the crawler that powers the assistant’s search. No, if you only block the one that collects text to train models. The whole setting comes down to that distinction, and many robots.txt files get it wrong.

OpenAI puts it plainly: a site that opts out of OAI-SearchBot will not be shown in ChatGPT search answers^[1]. GPTBot is for training: blocking it says your pages should not be used to train models, and it does not remove you from answers. Anthropic draws the same line between Claude-SearchBot, which indexes for search, and ClaudeBot, which collects for training^[2]. Perplexity recommends allowing PerplexityBot so your site appears in its results^[3].

At Google, Google-Extended is not a crawler but a control token. It decides whether your pages are used to train Gemini and ground its answers, and it does not affect your inclusion in Google Search^[4]. AI Overviews depend on Googlebot and regular indexing.

Which crawlers should you let in?

At a minimum, let in the assistants’ search crawlers: OAI-SearchBot, PerplexityBot, Claude-SearchBot, and Googlebot as always. Training crawlers are a publishing decision: blocking them protects your text and has no effect on today’s answers.

The table below lists the crawlers that iPand’s technical analysis checks first, with their role as their operator describes it. User-triggered agents such as ChatGPT-User are a separate case: they open a page because someone asked for it, and OpenAI warns that robots.txt rules may not apply to them^[1].

List maintained by iPand from the operators’ own pages, cited in the sources.

How to check your robots.txt in two minutes

Open your site’s address followed by /robots.txt in a browser and look for two things: a “Disallow: /” line under “User-agent: *”, and any group that names an AI crawler. If the file does not exist, everything is allowed by default.

Reading it takes one rule. A crawler follows the group that names it, and only that group; it falls back to the “*” group only when no group names it^[5]. So a “User-agent: OAI-SearchBot” group with no rules lets that crawler read everything, even if the “*” group blocks part of the site. And a “Disallow: /” under “*” blocks every crawler you did not name.

On WordPress, Shopify or Wix, the file is often generated by the CMS or an SEO plugin rather than uploaded by hand. Fix it where it is generated, or your change will be overwritten at the next update. A security plugin can also add rules without telling you: read the file as it is served, not as you think you wrote it.

When two rules in the same group conflict, the longest one wins: “Allow: /blog/” beats “Disallow: /” for your blog pages^[5].

• “Disallow: /” under “User-agent: *”: the whole site is closed to unnamed crawlers.
• GPTBot or ClaudeBot blocked: no effect on answers; it is a decision about training.
• OAI-SearchBot, PerplexityBot or Claude-SearchBot blocked: you drop out of their answers.
• No “Sitemap:” line: add your sitemap address, it helps crawlers find your pages.

The firewall and CDN trap

A perfect robots.txt does nothing if your host blocks crawlers before they ever read it. This is the most common case of a site that is “allowed” yet missing from answers.

Since July 1, 2025, Cloudflare asks every new domain at sign-up whether to allow AI crawlers^[6]. A setting accepted without much thought can block OAI-SearchBot or PerplexityBot at the network level, whatever your robots.txt says. Perplexity also notes that firewall rule changes can take time to propagate^[3].

So check two places: the robots.txt file, and the bot management or firewall rules of your host or CDN. The robots.txt page of our documentation explains what the technical analysis checks.

A sample configuration

The most common setup opens assistant search and leaves training up to you. Here is a file that allows the search crawlers, blocks two training crawlers and closes an admin folder to everyone.

# Assistant search crawlers: allowed
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
Disallow: /admin/

# Model training: refused (your call)
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

# All other crawlers
User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

Note that “Disallow: /admin/” is repeated in the first group: a named crawler follows only its own group and never reads the “*” one. After a change, allow about 24 hours for OpenAI and Perplexity to pick it up, according to their own pages^[1][3].

After the fix: check the effect

Opening the door does not guarantee a mention: it only makes you eligible. An assistant then chooses among the pages it can read, based on the question and the sources it cross-checks.

Write down the date of your fix, then follow your answers over several weeks. If you are still missing after opening up, the cause lies elsewhere: content that is too generic, an incomplete business profile, or competitors that are better represented in the sources assistants cite. Our articles on the business profile and on customer reviews cover those two levers.

What about llms.txt?

You do not need llms.txt to appear in AI answers. It is a proposal, published in September 2024, for a file that summarises a site for AI agents^[7], and no engine requires it.

Google says so for AI Overviews and AI Mode: there are no additional requirements, no special text file for AI and no special markup; a page needs to be indexed and eligible to show a snippet^[8]. If you publish an llms.txt, do it to help agents navigate, not in the hope of more visibility. robots.txt is still the front door.

To find out whether assistants mention you once the doors are open, run the free test on your own questions, then follow the visibility metrics over time.

Sources

1. OpenAI — Overview of OpenAI Crawlers — developers.openai.com/api/docs/bots , accessed October 23, 2026
2. Anthropic — crawling and blocking the crawler — support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler , accessed October 23, 2026
3. Perplexity — Perplexity Crawlers — docs.perplexity.ai/guides/bots , accessed October 23, 2026
4. Google Search Central — Google’s common crawlers — developers.google.com/search/docs/crawling-indexing/google-common-crawlers , accessed October 23, 2026, page dated October 15, 2026
5. IETF — RFC 9309, Robots Exclusion Protocol — www.rfc-editor.org/rfc/rfc9309.html , accessed October 23, 2026
6. Cloudflare — press release on AI crawlers — www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/ , accessed October 23, 2026, page dated July 1, 2025
7. llmstxt.org — the llms.txt proposal — llmstxt.org/ , accessed October 23, 2026, page dated September 3, 2024
8. Google Search Central — AI features and your website — developers.google.com/search/docs/appearance/ai-features , accessed October 23, 2026, page dated December 10, 2025