Answers / Getting cited
Should I block GPTBot? (And the AI crawlers you're blocking by accident)
Blocking GPTBot only opts you out of OpenAI's model training — it does not remove you from ChatGPT search. That's a separate bot (OAI-SearchBot) with an independent control, per OpenAI's own documentation. The bigger risk runs the other way: many sites blocked every AI crawler in 2023–24 on principle and are now invisible in AI answers as a direct result.
What does blocking GPTBot actually do?
It removes your content from the pool OpenAI uses to train its models — and nothing else. OpenAI's crawler documentation describes three separate agents with independent settings: GPTBot is the training crawler, OAI-SearchBot is what surfaces sites in ChatGPT search, and ChatGPT-User handles fetches a user directly asks for [1]. Disallowing one says nothing to the others.
The confusion is understandable — all three arrive from OpenAI, and in 2023 GPTBot was the only well-known name. But the documented consequences are almost opposite. Block GPTBot and your pages stay visible in ChatGPT search answers. Opt out of OAI-SearchBot and, in OpenAI's own words, your site "will not be shown in ChatGPT search answers" [1]. Per the same documentation, robots.txt changes take roughly 24 hours to take effect — so both mistakes and fixes propagate on about a one-day delay.
One more wrinkle: OpenAI's documentation notes that robots.txt rules "may not apply" to ChatGPT-User, since it acts on behalf of a human asking for a specific page [1]. A late-2025 revision of the documentation narrowed the robots.txt compliance language to GPTBot and OAI-SearchBot only [4]. Robots.txt governs the crawlers; it is not a fence around individual user requests.
Which bot controls what?
This is the control surface, drawn from the primary documentation wherever it exists. The short version: training controls and visibility controls are different levers, and the visibility levers are the ones businesses pull by accident.
| Crawler | What it's for | What blocking it does |
|---|---|---|
| GPTBot | OpenAI — training its models [1]. | Opts your content out of training. Does not affect ChatGPT search visibility. Respects robots.txt; changes apply in ~24h [1] [5]. |
| OAI-SearchBot | OpenAI — surfaces sites in ChatGPT search [1]. | Removes you: OpenAI's docs state opted-out sites "will not be shown in ChatGPT search answers." ~24h to propagate [1]. |
| ChatGPT-User | OpenAI — fetches a page when a user directly asks [1]. | Unreliable: OpenAI's docs say robots.txt "may not apply" here; the late-2025 doc revision dropped it from the compliance language [1] [4]. |
| Googlebot | Google — Search and AI Overviews, which are built on the Search index. There is no separate AI Overviews bot [2]. | Removes you from Google Search and its AI features together. To limit AI display without leaving Search, Google's docs point to nosnippet, max-snippet, and noindex [2]. |
| Google-Extended | Google — Gemini training and grounding only [3]. | Opts out of Gemini training/grounding. Google's docs state it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" [3]. |
| PerplexityBot | Perplexity — the crawler mapped to visibility in Perplexity answers [6]. | Cuts off the corresponding answer surface. |
| ClaudeBot | Anthropic — the crawler behind Claude. | Not covered by the primary documentation we cite here; check Anthropic's own docs before deciding. Treat it as one more independent token, not part of a blanket rule. |
So should you block GPTBot or not?
It depends on what you're protecting, and the documented controls let you split the difference.
- If you want AI assistants to send you customers: allow the answer-surface bots — OAI-SearchBot, Googlebot, PerplexityBot. These are the crawlers that determine whether you appear when a buyer asks. Blocking them is opting out of the channel.
- If your concern is content licensing or training use: the documented middle ground is to block the training crawlers — GPTBot and Google-Extended — while allowing the search bots. OpenAI documents these as independent settings [1], and Google states Google-Extended has no effect on Search inclusion or ranking [3]. You give up training use without giving up visibility.
- Never block Googlebot as an "AI block." AI Overviews has no separate crawler — blocking Googlebot takes you out of Google Search itself. If you want to limit how your content appears in Google's AI features, the controls are nosnippet, max-snippet, and noindex, not a robots.txt disallow [2].
Which AI crawlers are you blocking by accident?
Possibly several, and possibly without a single line in your robots.txt saying so. Two failure modes show up again and again:
The 2023–24 blanket block. When GPTBot was announced, blocking every AI user agent was a common defensive default. As one industry guide puts it, many sites blocked all AI crawlers in 2023–2024 on principle — and are now invisible in AI answers as a direct result [7]. Those rules rarely get revisited, and the newer search-surface bots often inherited the block. If your robots.txt predates OAI-SearchBot's existence, it's worth rereading with the table above open.
Blocks robots.txt doesn't reveal. Robots.txt is only the polite, visible layer. Firewall and bot-protection rules at your WAF or CDN can turn away AI crawlers before they ever read robots.txt — and nothing on the page tells you it's happening. The only way to know is to test the answers themselves: run your buyers' real questions through the engines and see whether you exist there. Our free self-check walks through exactly that.
Sources
- OpenAI — crawler and bot documentation (primary): GPTBot, OAI-SearchBot, and ChatGPT-User roles, independent controls, opt-out consequences, ~24h robots.txt propagation.
- Google Search Central — AI features documentation (primary): AI Overviews built on the Search index; snippet and indexing controls.
- Google — common crawlers documentation (primary): Google-Extended scope and its non-effect on Search inclusion and ranking.
- PPC Land — OpenAI revises ChatGPT crawler documentation: the late-2025 narrowing of robots.txt compliance language.
- Search Insight — Does ChatGPT actually obey your robots.txt rules?
- LLM Pulse — AEO vs GEO vs SEO: crawler-per-surface mapping.
- Searchpod — How do I get my business recommended by ChatGPT?: the 2023–24 blanket-block pattern and its consequences.