AI Crawlers & User Agents

You spent months earning content worth citing, and then quietly blocked the very bots that would carry it into AI answers. It happens more than anyone admits, often at the CDN, where a single toggle can shut the harbour gates without you ever seeing it. This page lists the AI crawlers that matter in 2026, what each one does, and how to make sure they can actually reach you.

The short version

This is part of our AI SEO guide. Getting crawler access right is the most fundamental AI SEO step there is, because everything else depends on the bots being able to read you in the first place.

11%

of domains are cited by both ChatGPT and Perplexity

Each engine reads via its own crawlers. 5W PR, May 2026 synthesis

84%

of AI citations come from earned media

Fed to the engines through the same crawlers you keep open. Muck Rack, May 2026

0

answers from a bot you turned away

A closed gate guarantees failure. Opening it is step zero, the one nothing else can replace.

Different bots do different jobs, and the distinction matters for what you allow. Broadly, there are training crawlers (gathering content to train models) and search or answer crawlers (fetching pages to answer a live question, often with a citation back to you). For visibility, the answer crawlers are the ones you most want to welcome, because they’re the ones that can send a citation your way right now.

OpenAI

OpenAI runs GPTBot, used to gather content that may train its models, and OAI-SearchBot, tied to ChatGPT’s search results and the citations that appear there. There’s also ChatGPT-User, which fetches a page when a user or a custom action specifically requests it. Allowing the search-related agents is what helps you appear in ChatGPT’s answers.

Anthropic

Anthropic operates ClaudeBot for content gathering, alongside agents tied to Claude’s search and live answers such as Claude-SearchBot and the anthropic-ai user agent. Allowing these lets your pages inform Claude’s responses.

Perplexity

Perplexity runs PerplexityBot, which crawls for its index, and Perplexity-User, which fetches a page in response to a specific user query. Since Perplexity is one of the most citation-heavy engines, keeping these unblocked is high value.

Google and Microsoft

Google uses Google-Extended as a control specifically for whether your content can be used for Gemini and AI training, separate from regular Googlebot. Microsoft’s Bingbot powers both Bing search and Copilot’s web answers, so blocking it costs you on both. Note a subtlety: blocking Google-Extended doesn’t remove you from normal Google Search, only from the AI-training use, which is a genuine choice to make rather than an accident to avoid.

Training crawlers

Like GPTBot in its training role and ClaudeBot, they gather content that may go into the next version of a model. The benefit to you is indirect and slow: being in the training data means a model might “know” your brand later, with no link and no clear attribution. Some publishers are wary of this, since it uses their work to build a product they don’t control.

Answer crawlers

Like OAI-SearchBot, PerplexityBot, and Claude-SearchBot, they fetch your page to answer a live question, and they typically cite the source. The benefit here is direct and immediate: a citation in an answer, often with a link, that can send a real visitor or plant your name in a buyer’s mind. For most brands chasing AI visibility, these are the bots that matter, and the case for allowing them is strong and uncomplicated.

The practical line

If you’re going to be selective, lean toward welcoming the answer crawlers without hesitation, and treat the training crawlers as the more personal call. Blocking answer crawlers to protect against training is the worst of both worlds, since it costs you the visible upside to guard against the invisible one.

How to allow the right bots

The mechanism is your robots.txt file. To welcome the AI crawlers, make sure none of them are disallowed, and ideally add explicit Allow lines for the ones you want so there’s no ambiguity.

• Add explicit Allow rules for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, and Bingbot.
• Check your global rules. A broad Disallow under User-agent: * can accidentally catch bots you meant to allow. Bot-specific Allow rules should sit above it.
• Decide on training deliberately. If you want the visibility but not the training use, you can allow the search agents and disallow the pure training crawlers. That’s a real choice; just make it on purpose, not by accident.
• Consider an llms.txt file too. Alongside robots.txt, a llms.txt file can point AI engines at the content you most want them to read.

One reason teams block aggressively is that user agents can be faked: a scraper can claim to be GPTBot to slip past defences. That’s a real concern, but blanket-blocking the AI crawlers is a blunt response that throws away your visibility to stop impersonators. The better approach is verification, since the major AI companies publish IP ranges or verification methods for their official crawlers.

This is the one that catches the most people, so it gets its own section. Content delivery networks and security services often ship with AI-bot blocking available as a simple switch, and on some plans it can be on by default or turned on by a well-meaning teammate worried about scraping. The result is brutal and quiet: your robots.txt cheerfully invites the AI crawlers, and the edge turns them away before they ever see it.

Verify, don’t blanket-block

The major AI companies publish IP ranges or verification methods for their official crawlers, so you can confirm a request claiming to be PerplexityBot or GPTBot really comes from them, and challenge the rest. That keeps the visibility while screening out impersonators.

The pragmatic default

If verification is more than you want to manage, still allow the named AI user agents and accept a small amount of impersonation as the cost of being visible. For most brands, a few fake bots are a far smaller problem than disappearing from every AI answer. Decide based on your actual risk, and revisit it if the impersonation ever becomes a measurable load.