How to configure robots.txt for AI crawlers: GPTBot, ClaudeBot and PerplexityBot

The quick answer

To configure robots.txt for AI crawlers, add a User-agent block for each named bot (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot) and set Allow: / to permit crawling or Disallow: / to block it, then place the file at your domain root (https://example.com/robots.txt). Blocking an AI crawler removes your pages from that engine's training and retrieval corpus, which usually lowers your chance of being cited in ChatGPT, Claude, Perplexity or Gemini answers. As of 2026, this guide lists the canonical user-agent strings for each major AI crawler, the allow and block directives, the citation tradeoff of blocking, and a copy-paste robots.txt template.

Note on data: AI crawler operators rename and split user agents over time (OpenAI alone runs three distinct bots). The strings below are accurate as of 2026; verify current user agents on each vendor's official crawler documentation before relying on a directive.

What robots.txt controls for AI crawlers (and what it does not)

Robots.txt is a plain-text file at a domain root that tells crawlers which paths they may fetch, using User-agent lines to name a bot and Allow or Disallow lines to permit or forbid paths. As of 2026, robots.txt governs three different AI crawler jobs (training data collection, live retrieval for answers, and search indexing), and a single directive can hit one job without affecting the others.

Robots.txt is a voluntary standard (the Robots Exclusion Protocol), so it depends on each crawler choosing to obey it. OpenAI, Anthropic, Perplexity and Google all publish that their named crawlers respect robots.txt as of 2026.

What a robots.txt directive does and does not do:

• Does control fetching: a Disallow: / under User-agent: GPTBot instructs GPTBot to stop fetching every path on the host. OpenAI documents that GPTBot honors this directive.
• Does not retroactively delete data: blocking GPTBot in 2026 stops future fetches but does not remove pages already ingested into a prior training snapshot.
• Does not block one engine by blocking another: blocking GPTBot (OpenAI training) does not block ClaudeBot (Anthropic) or PerplexityBot (Perplexity), because each operator runs a separately named crawler.
• Does not stop a determined non-compliant bot: robots.txt is an instruction, not a firewall. A crawler that ignores the protocol still fetches the page; enforcement requires server-side blocking (WAF, IP rules).

For the underlying concept of how AI engines surface and cite your pages, see AI search and AI citations.

The canonical AI crawler user agents to name in robots.txt

AI crawlers are identified in robots.txt by their exact user-agent token, which must be spelled and cased as the operator publishes it, because robots.txt user-agent matching is case-insensitive on the token but ignores any block whose token it cannot match. As of 2026, the table below lists the canonical user-agent strings for the major AI crawlers and the job each one performs.

Three distinctions decide your config. First, OpenAI splits its crawler into three tokens (GPTBot for training, OAI-SearchBot for search indexing, ChatGPT-User for live user-triggered fetches), so blocking GPTBot alone still lets ChatGPT retrieve your page live. Second, Google-Extended controls Gemini training and grounding but does not control Googlebot, so blocking Google-Extended keeps you in Google Search and AI Overviews while opting out of Gemini training. Third, CCBot (Common Crawl) feeds many third-party models indirectly, so a model you never named can still train on your pages through the Common Crawl dataset.

Allow or block: the directive syntax for each AI crawler

A robots.txt block pairs one User-agent token with one or more Allow or Disallow path rules, and the most specific matching rule wins for a given path. As of 2026, allowing an AI crawler requires either no Disallow rule for that token or an explicit Allow: /, while blocking requires Disallow: / under the exact token.

To allow every AI crawler (default for maximum citation eligibility), name no AI bot with a Disallow, or be explicit:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

To block one AI crawler while allowing the rest, target only that token:

User-agent: GPTBot
Disallow: /

To block all AI training but keep live retrieval and search, block the training bots and leave the user-fetch bots allowed:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

To block a single sensitive path for an AI crawler while allowing the rest of the site, use a path-scoped Disallow:

User-agent: GPTBot
Disallow: /private/
Allow: /

Two syntax rules prevent silent failures. First, the User-agent token must match exactly (GPTBot, not GptBot spelled with a missing capital intent, not OpenAI); an unrecognized token means the block is ignored and the bot falls back to the wildcard User-agent: * rule. Second, a wildcard User-agent: * with Disallow: / blocks every crawler including Googlebot, so never use the wildcard to target only AI bots.

The citation tradeoff: what blocking AI crawlers costs you

Blocking an AI crawler in robots.txt removes your pages from that engine's retrieval and training corpus, which directly lowers your eligibility to be cited in that engine's answers. As of 2026, an engine cannot cite a source it was instructed not to fetch, so a Disallow: / under PerplexityBot makes Perplexity citations of your domain effectively impossible.

The tradeoff splits by crawler job:

• Blocking training bots (GPTBot, ClaudeBot, CCBot, Google-Extended) keeps your content out of future model weights but does not remove pages already in a 2026 or earlier training snapshot. The cost is reduced presence in parametric (memorized) answers over time.
• Blocking retrieval and search bots (OAI-SearchBot, PerplexityBot, ChatGPT-User) is the larger citation cost, because these bots feed the live, cited answers users see today. Blocking them removes you from the source list that ChatGPT search and Perplexity attach to answers.
• Blocking Googlebot is the most expensive directive: it removes you from Google Search, Google AI Overviews and AI Mode at once, so never use it to manage AI exposure.

The decision is rarely all-or-nothing. Publishers with paid or proprietary content often block training bots (GPTBot, CCBot) to protect the corpus while allowing retrieval bots (OAI-SearchBot, PerplexityBot) so the brand still earns cited answers. Most B2B SaaS and marketing sites that want AI visibility should allow all named crawlers, because every block narrows the set of engines that can cite the page. For why citation eligibility matters more than raw rankings, see why your Google rankings do not translate into AI citations and what GEO is.

A copy-paste robots.txt template for AI crawlers (2026)

This template allows all major AI crawlers (maximum citation eligibility) while keeping classic search intact, and includes commented block lines you can uncomment to restrict training only. As of 2026, place the file at https://yourdomain.com/robots.txt and serve it with a 200 status and text/plain content type.

# robots.txt: AI crawler configuration (updated 2026)
# Default: allow all AI crawlers for maximum citation eligibility.

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

# Keep classic search open (do not block Googlebot to manage AI).
User-agent: Googlebot
Allow: /

# Wildcard fallback for all other bots.
User-agent: *
Allow: /

# --- To block AI training only, comment out the Allow blocks above
# --- for these tokens and uncomment the lines below:
# User-agent: GPTBot
# Disallow: /
# User-agent: ClaudeBot
# Disallow: /
# User-agent: CCBot
# Disallow: /
# User-agent: Google-Extended
# Disallow: /

Sitemap: https://yourdomain.com/sitemap.xml

Three deployment checks prevent a dead file. First, robots.txt must sit at the host root, not a subfolder; a file at /blog/robots.txt is ignored. Second, each subdomain needs its own robots.txt (blog.example.com/robots.txt is separate from example.com/robots.txt). Third, a Sitemap: line is global, not tied to any User-agent block, and helps every compliant crawler discover your pages.

How to verify AI crawlers are obeying your robots.txt

Verifying AI crawler behavior requires reading your server access logs for each named user agent and confirming the fetch pattern matches your robots.txt directive, because robots.txt is advisory and only logs prove compliance. As of 2026, a compliant crawler that you allowed appears in logs fetching pages, while a crawler you blocked should stop appearing after its next robots.txt re-read.

Verification steps in order:

1. Grep your access logs for each user-agent token: search for GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and Google-Extended in the user-agent field of your web server or CDN logs.
2. Confirm allowed bots fetch: an allowed GPTBot should appear fetching multiple paths with 200 responses. Zero GPTBot hits over weeks can mean a misconfigured block or simply low crawl priority.
3. Confirm blocked bots stop: after a Disallow: /, a compliant crawler re-reads robots.txt (often within 24 hours) then stops fetching. Continued fetches indicate either a caching delay or a non-compliant bot.
4. Watch for non-compliant fetchers: a bot ignoring Disallow requires server-side enforcement (WAF rule, IP block, or a CDN bot-management ruleset), because robots.txt cannot stop it.

Logs answer "did the crawler fetch my page", not "did the engine cite my page". A crawler can fetch your content and still not cite it in an answer, because fetching is necessary but not sufficient for citation. To measure the second question (which engines actually cite your domain, per query and per country), you need on-engine citation data rather than crawl logs; see how to measure AI share of voice.

From crawl access to citation: closing the loop

Allowing AI crawlers in robots.txt makes a page eligible to be cited, but eligibility is not citation, so the next question is which engines actually cite your domain and which competing sources they cite instead. As of 2026, robots.txt is the permission layer and on-engine citation tracking is the measurement layer, and the two answer different questions.

How the two layers differ:

Getspotted is the citation layer, not a crawler config tool: one /search call returns the sources Google, ChatGPT, AI Overview, Perplexity, Claude and Gemini cite for a query, per country, plus the contacts behind each cited source. Used after you have allowed the AI crawlers above, robots.txt opens the door and Getspotted reports who walked through it. Browse which sources AI cites for live examples, or read what generative engine optimization is for the strategy that turns crawl access into citations.

If you are a developer wiring citation data into your own monitor or AI agent, the API docs show the /search and contact endpoints. If you want a faster path to being cited once crawlers can reach you, see how to get cited in AI answers.

FAQ

How do I allow GPTBot in robots.txt?

To allow GPTBot, add a User-agent: GPTBot block with Allow: / (or simply name no Disallow for GPTBot), then place robots.txt at your domain root. As of 2026, OpenAI documents that GPTBot honors robots.txt, so an Allow: / directive permits GPTBot to fetch every path for GPT training data collection.

What is the ClaudeBot user agent string?

ClaudeBot is Anthropic's crawler token used for training and retrieval, and Claude-Web is the separate token for live browsing fetches. As of 2026, name ClaudeBot exactly (case-insensitive on the token) in a User-agent: ClaudeBot block; an unrecognized spelling makes the block fall back to the wildcard User-agent: * rule.

How do I give PerplexityBot crawl access?

Give PerplexityBot crawl access with a User-agent: PerplexityBot block and Allow: /, and allow Perplexity-User too if you want live user-initiated fetches. As of 2026, PerplexityBot indexes pages for Perplexity answers, so allowing it is required for your domain to appear in Perplexity's cited source list.

Should I block or allow AI crawlers?

Most B2B SaaS and marketing sites that want AI visibility should allow all named AI crawlers, because every block removes the domain from that engine's citable corpus. Block AI crawlers only to protect paid or proprietary content; in that case block training bots (GPTBot, CCBot) while allowing retrieval bots (OAI-SearchBot, PerplexityBot) to keep earning cited answers.

What does Google-Extended do in robots.txt?

Google-Extended controls whether Google uses your pages for Gemini training and grounding, separately from Googlebot which controls Search and AI Overviews indexing. As of 2026, a Disallow: / under Google-Extended opts you out of Gemini training while keeping you in Google Search and AI Overviews, because the two tokens govern different jobs.

Does blocking AI crawlers remove my content from ChatGPT?

Blocking AI crawlers in robots.txt stops future fetches but does not remove content already ingested into a prior training snapshot. As of 2026, a Disallow: / under GPTBot prevents new GPT training fetches, yet pages crawled before the block can remain in earlier model weights until that model is retrained.

Where does robots.txt for AI crawlers need to be placed?

robots.txt must sit at the host root, served as https://yourdomain.com/robots.txt with a 200 status and text/plain content type. As of 2026, a file in a subfolder (/blog/robots.txt) is ignored, and each subdomain needs its own robots.txt, because crawlers fetch the file only from the exact host root they are crawling.

Can robots.txt fully stop a non-compliant AI bot?

No, robots.txt is an advisory instruction, not a firewall, so a crawler that ignores the Robots Exclusion Protocol still fetches your pages. As of 2026, stopping a non-compliant AI bot requires server-side enforcement (a WAF rule, IP block, or CDN bot-management ruleset); robots.txt only governs crawlers that choose to obey it.