The Blocked from AI Search feature in Site Audit shows how many of the pages checked in your audit are currently blocked from being accessed by AI search engines. If these bots can’t crawl your site, your content might not appear in AI-generated search results.
You’ll find it in the Crawled Pages report.
This feature is designed to help you:
- Understand which AI bots are blocked by a domain’s robots.txt file
- Decide whether to allow or disallow specific AI crawlers
- Optimize your site’s visibility in AI-powered search engines
Why It Matters
Modern AI assistants and search engines rely on specialized bots to fetch and index web content. Blocking these bots may limit your content’s appearance in:
- ChatGPT responses
- Google AI Overviews
- Perplexity search results
- Claude and other AI search tools
In short: Disallowed bots = missed opportunities for AI visibility.
Types of Bots Related to AI Search
There are four main types of bots used by AI search platforms to crawl websites and gather information.
- AI assistant bots / on-demand fetcher bots
- AI search crawlers
- AI training bots
- Traditional search bots
1. AI Assistant Bots / On-Demand Fetcher Bots
Bots that fetch content in real time when a user submits a query in an AI assistant like ChatGPT or Claude.
Examples:
- ChatGPT-User
- Claude-User
- Perplexity-User
Why it matters:
Blocking these bots can prevent your content from being included in conversational answers.
2. AI Search Crawlers
Bots that crawl and index your site content ahead of time, to power AI search engines (like ChatGPT’s search mode or Perplexity’s search engine).
Examples:
- OAI-SearchBot
- Claude-SearchBot
- PerplexityBot
- Googlebot (used for both classic and AI search results like AI Overviews and AI Mode)
Why it matters:
These bots are central to visibility in AI-driven search results. Blocking them can limit your site’s reach.
3. AI Training Bots
Bots that scrape content at scale for training large language models. These do not affect your search visibility — but may raise content ownership concerns.
Examples:
- Google-Extended
- GPTBot
- ClaudeBot
Note:
Google-Extended is included here because its public description can be misleading. It’s often mistaken for a search crawler, but in reality, it’s an AI training bot used by Google to feed its generative models. The same applies to GPTBot (OpenAI) and ClaudeBot (Anthropic).
4. Traditional Search Bots
Classic SEO crawlers like Googlebot or Bingbot follow instructions found in robots.txt, and blocking them affects your rankings in traditional search engines, which can indirectly impact visibility in AI search.
What Site Audit Checks For
The Blocked from AI Search check in your Site Audit dashboard shows how many pages from those checked in your audit are blocked from any specific AI search engine bots via robots.txt.
Use this data to:
- Check if any AI bots are unintentionally blocked
- Identify risks to your AI search visibility
- Adjust access permissions based on your visibility goals
AI Crawlers/Bots Semrush Site Audit Checks For
|
AI Bot |
What it does |
Type |
Source |
|
ChatGPT-User |
Interacts with websites to gather data when a user starts a query in ChatGPT. |
On‑Demand Fetcher Bot |
|
|
OAI-SearchBot |
OpenAI’s crawler for indexing websites into ChatGPT/SearchGPT search results. |
AI Search Crawler |
|
|
Googlebot |
Google’s main web crawler used to index content for Search and integrated AI features (AI Overview, Google AI Mode). |
Traditional SEO bot, AI Search Crawler |
|
|
Google-Extended |
Robots.txt token that allows users to control whether Google can use their crawled content for training Gemini AI models. |
AI Training Bots, AI Search Crawler |
|
|
Perplexity‑User |
Interacts with websites to gather data when a user starts a query in Perplexity AI. |
On‑Demand Fetcher Bot |
|
|
PerplexityBot |
Crawler used to index sites for Perplexity AI’s search engine. |
AI Search Crawler |
|
|
Claude-User |
Interacts with websites to gather data when a user starts a query in Claude. |
On‑Demand Fetcher Bot |
|
|
Claude-SearchBot |
Anthropic’s crawler for indexing websites into Claude search results. |
AI Search Crawler |
Should You Allow AI Search Bots?
In most cases, if you want your content to appear in AI-powered search engines, yes. You need to allow their bots to access your site.
Benefits of allowing AI search bots to access your site:
- Better visibility in AI responses
- Potential for more traffic from users using AI assistants
- Future-proofing your AI SEO strategy to allow for AI-powered experiences
Robots.txt Examples: How to Allow AI Search Bots
To allow access, paste the following lines into your robots.txt file. These lines grant permission to key AI bots grouped by provider.
# — OPENAI —
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
# — GOOGLE —
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Allow: /
# — PERPLEXITY —
User-agent: Perplexity-User
Allow: /
User-agent: PerplexityBot
Allow: /
# — ANTHROPIC —
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
Tip: If you're using a firewall or bot management platform, make sure these user-agent names are also whitelisted there.
Summary: Key Takeaways
- AI bots power modern search experiences. Blocking them can limit your visibility.
- The Site Audit feature shows which of your pages are blocked from AI search bots.
- You can control access using robots.txt, and we recommend allowing trusted bots to improve your chances of appearing in AI search results.