In Brief
- Core Answer: AI crawlers are the bots that fetch web content for language model training, for AI search indexes, and for live user-initiated retrieval. They are distinct user agents with distinct purposes, and blocking the wrong one removes you from an answer surface entirely.
- Why It Matters: The most common cause of a brand being invisible in one assistant and visible in another is a robots.txt rule nobody reviewed, usually added by a security or bandwidth decision.
- Best For: Technical SEO owners and anyone auditing why their content is not being cited.
AI crawlers fall into three functional groups: training crawlers that collect data for future model versions, search crawlers that build the index an assistant retrieves from, and user agents that fetch a specific page in response to a live question. The distinction matters because blocking a training crawler is a licensing decision, while blocking a search crawler is a visibility decision — and many sites make the second by accident while intending the first.
The Three Functional Types
Type | Purpose | Effect of blocking |
|---|---|---|
Training crawler | Collects text for future model training | No effect on current answers; a content-licensing choice |
Search crawler | Builds the retrieval index the assistant queries | Removes you from that assistant's answers |
User agent | Fetches a page a user or assistant explicitly asked for | Breaks live lookups of your pages |
A defensible default for most commercial sites: allow search crawlers and user agents, and make a deliberate, documented decision about training crawlers. Publishers with licensing revenue at stake reasonably choose differently.
The AI Crawlers That Matter
- Googlebot — the main search crawler. This is what feeds AI Overviews and AI Mode. Blocking it removes you from Google entirely, AI surfaces included.
- Google-Extended — a control token, not a separate crawler. It governs whether your content is used to improve Gemini and Vertex AI generative models. Google's documentation is explicit that it does not affect inclusion in Google Search, and therefore does not affect AI Overviews. This is the single most misunderstood entry in the list: disallowing Google-Extended is a training opt-out, not a visibility decision.
- GoogleOther — used for various non-search fetches such as research and product development.
OpenAI
- GPTBot — training crawler. Blocking is a licensing decision with no effect on ChatGPT search results.
- OAI-SearchBot — builds the index used by ChatGPT search. Blocking this removes you from ChatGPT's search-grounded answers. This is the one to check first.
- ChatGPT-User — fetches a specific page when a user or a tool call requests it live. Blocking breaks direct lookups of your pages.
Anthropic
- ClaudeBot — training crawler.
- Claude-SearchBot — supports search-grounded answers.
- Claude-User — live, user-initiated fetches.
Perplexity
- PerplexityBot — indexes content for Perplexity's search results and citations.
- Perplexity-User — fetches a page when a user follows or requests it directly.
Microsoft
- Bingbot — feeds Bing and, by extension, Copilot's search grounding. Several other assistants have historically used Bing's index too, so Bingbot's reach is wider than its name suggests.
Others Worth Knowing
- Applebot and Applebot-Extended — the second is a training opt-out token, mirroring Google's split.
- meta-externalagent — Meta's crawler for AI products.
- Amazonbot — Amazon's, including Alexa-related retrieval.
- CCBot — Common Crawl. Not an AI company, but its archives are a widely used training corpus, so blocking it has indirect long-term effects.
- Bytespider — ByteDance. Frequently blocked on bandwidth grounds rather than policy.
The list changes. Re-check the official documentation quarterly rather than relying on a blog post — including this one.
How to Audit Your Own Configuration
- Read your robots.txt as a machine would. Wildcard rules and ordering matter. A
User-agent: *block with a broad Disallow applies to every agent that lacks its own block, which routinely catches AI search crawlers nobody intended to exclude. - Check the layer above robots.txt. CDN bot-management rules, WAF policies, rate limits and geo-blocking are invisible in robots.txt and reject crawlers before your server sees them. This is where most accidental blocks actually live, and the marketing team never hears about the change.
- Grep your logs by user agent. Filter the last thirty days for each agent above and count requests and status codes. A crawler you believe is allowed but which receives 403s is being blocked somewhere you have not looked.
- Verify rendering. Several of these crawlers do not execute JavaScript, or execute it inconsistently. Content that only exists after client-side hydration is content they cannot read. Fetch a key page with curl and confirm the substance is in the HTML.
- Check status codes and redirects. Long redirect chains and soft 404s degrade crawl efficiency for every agent.
The Decision Framework
Three questions resolve most policy debates.
- Do you sell to people who research in assistants? If yes, search crawlers and user agents must be allowed. There is no version of AI visibility that survives blocking the index that answers are built from.
- Is your content itself the product? Publishers, research firms and course providers have a genuine argument for restricting training crawlers, because the content has licensing value that free ingestion erodes. Software and services companies usually do not — their content is marketing, and being ingested is the point.
- Is the concern actually bandwidth? If crawler volume is a cost problem, rate-limit rather than block. A 429 with a sensible retry policy preserves eventual access; a 403 is permanent exclusion.
What Blocking Does Not Achieve
Two limits are worth stating plainly. First, blocking a training crawler today does not remove content already ingested into an existing model — that content is in the weights and stays there until the model is replaced. Second, robots.txt is voluntary. Compliant crawlers honour it; non-compliant scrapers do not, and content republished elsewhere is reachable regardless of what your file says.
The realistic goal of a crawler policy is therefore to shape future ingestion and to keep the surfaces you want to appear in open. It is not a content-protection mechanism.
A Sensible Default Configuration
For a B2B software or services company with no content-licensing revenue, the configuration that maximises visibility and preserves optionality is: allow Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot and Claude-User without restriction; make an explicit, minuted decision on GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot; and rate-limit rather than block anything creating genuine load. Then re-audit quarterly, because both the agent list and your own infrastructure rules will have changed.
Once access is confirmed, the constraint moves to whether your content is worth quoting — which is a content and consensus problem, not a technical one.
Reading Crawler Traffic in Your Logs
Once access is configured, the logs become a genuinely useful measurement surface — and one almost nobody looks at. Four things worth extracting monthly:
- Requests per agent, per week. A sudden drop for one agent usually means an infrastructure change on your side, not a policy change on theirs. Catching it in the logs is faster than noticing your citations fell two months later.
- Which URLs each agent fetches. Search crawlers spread across the site; user agents concentrate on specific pages, and those pages are the ones being surfaced in live answers. That list is a free signal about which content is actually working.
- Status code distribution. A crawler receiving mostly 200s is healthy. A high proportion of 403s, 429s or 5xx responses means you are being throttled, blocked or timing out — all of which reduce inclusion without any visible error anywhere else.
- Response time. Crawlers have budgets. Slow responses reduce crawl depth, which quietly limits how much of a large site is available to be retrieved.
Note that crawler traffic is not audience traffic and should never be reported as such. Bot volume can be substantial on a content-heavy site, and mixing it into session counts inflates numbers that other teams will act on. Segment it out and report it separately, as infrastructure telemetry rather than marketing performance.
The Verification Problem
User-agent strings are trivially spoofed. Scrapers routinely identify themselves as GPTBot or PerplexityBot to obtain the more permissive treatment those agents receive, and a naive log analysis will count them as legitimate.
The major operators publish IP ranges or support reverse DNS verification for exactly this reason. If crawler access is a meaningful policy question for your organisation — because you are making a licensing decision, or because bandwidth costs are material — verify by IP rather than by user-agent string. For most companies this is more rigour than the decision warrants, but it is worth knowing that the string alone proves nothing.
The practical consequence for visibility work is small: a spoofed crawler still fetched your page, and your content is not more or less quotable either way. The consequence for a bandwidth or licensing decision is larger, because you may be blocking a real crawler in response to load generated by an impostor.
Anthropic's three crawlers deserve special attention because blocking the wrong one removes a site from Claude's answers; Claude SEO covers the split and why Claude referrals matter.
Frequently Asked Questions
What are AI crawlers?
AI crawlers are bots operated by AI companies that fetch web content for three purposes: training future models, building the search indexes assistants retrieve from, and fetching specific pages during a live user request.
Should I block GPTBot?
Blocking GPTBot is a training opt-out and does not affect whether you appear in ChatGPT's search-grounded answers, which use OAI-SearchBot. Block it only if your content has licensing value you want to protect.
Does blocking Google-Extended affect AI Overviews?
No. Google's documentation states Google-Extended governs use of content for improving Gemini and Vertex AI generative models and does not affect inclusion in Google Search. AI Overviews draw on the Search index built by Googlebot.
Which AI crawler should I definitely allow?
The search-grounding crawlers: Googlebot, Bingbot, OAI-SearchBot, PerplexityBot and Claude-SearchBot, plus the user agents that fetch pages on request. Blocking any of these removes you from that platform's answers.
How do I know if AI crawlers are reaching my site?
Filter your server logs by user agent for the last thirty days and check request counts and status codes. A crawler receiving 403 responses is being blocked somewhere — usually a CDN or WAF rule rather than robots.txt.
Do AI crawlers execute JavaScript?
Inconsistently, and several do not. Content that only appears after client-side rendering may be invisible to them. Server-render anything you want quoted, and verify by fetching the page without a browser.
References
- https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
- https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
- https://platform.openai.com/docs/bots
- https://docs.perplexity.ai/guides/bots
- https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler