- Which AI crawlers should I allow if I want citations in AI search engines?
- Allow search grounding bots like ChatGPT-User, PerplexityBot, and Claude-Web. Blocking them completely removes your website from AI search answer citations.
- What is the difference between GPTBot and ChatGPT-User?
- GPTBot crawls the web to train future OpenAI foundation models. ChatGPT-User executes live on-demand browsing when a user submits a query in ChatGPT.
- How does Google-Extended differ from standard Googlebot?
- Googlebot crawls for standard Google Search indexing. Google-Extended specifically controls whether your content is used to train Gemini models without hurting Google Search rankings.
- Can this tool generate corrected robots.txt rules for my website?
- Yes. The tool provides ready-to-paste robots.txt snippets tailored to your preference for search citations versus training protection.
- What is an AI crawler and how does it differ from a traditional search crawler?
- An AI crawler is an automated HTTP robot operated by artificial intelligence laboratories (such as OpenAI, Anthropic, Perplexity, or Google) designed to fetch web content. Unlike traditional search crawlers like Googlebot—which index pages primarily to rank links in a 10-blue-links search results page—AI crawlers retrieve text either to synthesize direct natural-language answers (e.g. OAI-SearchBot, PerplexityBot) or to build foundational training corpora (e.g. GPTBot, ClaudeBot).
- What is the critical difference between OAI-SearchBot and GPTBot?
- OAI-SearchBot is OpenAI's real-time search and citation crawler. It discovers web pages so that ChatGPT can cite them as live sources with hyperlinked footnotes during user queries. GPTBot is OpenAI's model training crawler used to collect public internet datasets for future model architectures. Allowing OAI-SearchBot enables live visibility in ChatGPT search, while blocking GPTBot prevents your content from being ingested into foundation model weights.
- Does blocking GPTBot or ClaudeBot hurt my visibility in ChatGPT or Claude search?
- No. Both OpenAI and Anthropic have explicitly separated their search discovery crawlers from their model training bots. Blocking GPTBot (User-agent: GPTBot) does not prevent OAI-SearchBot from surfacing your articles in ChatGPT search experiences. Similarly, blocking ClaudeBot does not restrict Claude-SearchBot or Claude-User.
- What is Google-Extended and does blocking it remove my site from Google Search?
- Google-Extended is a robots.txt product token created by Google that allows publishers to manage whether their content is used to train Gemini models and power vertex generative AI features. Crucially, blocking Google-Extended does NOT affect your website’s ranking or indexing in traditional Google Search, which is governed independently by Googlebot.
- Why can't I perform a fake HTTP request pretending to be Google-Extended?
- Google-Extended is not an independent HTTP User-Agent string sent during network connections; it is strictly a robots.txt directive token parsed by Google's backend crawl infrastructure. Testing it requires evaluating your robots.txt file for directives under "User-agent: Google-Extended", rather than sending an HTTP request with that header.
- What is the difference between automated indexing and user-initiated AI retrieval (Claude-User / Perplexity-User)?
- Automated crawlers (such as Claude-SearchBot or PerplexityBot) traverse the internet autonomously on a scheduled basis to populate search indices. User-initiated retrieval agents (such as Claude-User or Perplexity-User) only fetch a specific web page when an end-user explicitly enters a URL or clicks a link inside a conversational AI session asking for a live summary.