Technical Specification & Formula
Audited Agents: ClaudeBot • Claude-SearchBot • Claude-User • PerplexityBot • Perplexity-User • Google-Extended • Applebot-Extended
Step-by-Step: ClaudeBot & PerplexityBot Checker
- Paste Target URL. Enter the exact URL you want Anthropic and Perplexity crawlers to index and cite.
- Inspect Rule Precedence & Headers. Detect conflicting wildcard rules, meta noindex tags, or X-Robots-Tag noai directives.
- Verify HTTP 200 Reachability. Confirm your server responds cleanly without redirect loops or 403 blocks.
Frequently Asked Questions
- Which user-agent controls Claude web search citations?
- Anthropic uses Claude-SearchBot for search indexing, Claude-User for user-triggered fetches, and ClaudeBot for broader corpus crawling. Allowing Claude-SearchBot and Claude-User ensures Claude can cite your live pages.
- Does Cloudflare Bot Fight Mode block PerplexityBot and ClaudeBot?
- Yes, aggressive "Block AI Scrapers" toggles in Cloudflare or WAFs frequently return HTTP 403 to PerplexityBot and ClaudeBot even when robots.txt says Allow: /.
More AI Crawler Checker Workflows
- Full AI Crawler Checker Workspace — Audit your website's robots.txt directives and X-Robots-Tag headers for AI crawlers, search bots, and training spiders.
- GPTBot & OAI-SearchBot Checker — Test if GPTBot, OAI-SearchBot, and ChatGPT-User are allowed or blocked in your robots.txt and HTTP X-Robots-Tag headers. Free instant OpenAI crawler audit.
Related Free SEO Tools
- LLMs.txt Generator & Validator — Create and validate official llms.txt and llms-full.txt files according to the latest specification for AI agents, developer tooling, and LLM search indexing.
- AI Visibility Checker — Check whether ChatGPT, Gemini and Perplexity cite your website, mention your brand, or recommend competitors instead. Free, no sign-up, results in about a minute.
- AI Link Hallucination Checker — Detect hallucinated and broken URLs in AI-generated drafts. Automatically verifies HTTP status codes, DNS validity, and SSL certificates.
GrowthControl — Track AI Visibility in ChatGPT, Gemini, Perplexity & Claude →
Audit whether AI search engines mention and cite your brand with our free AI Brand Visibility Checker.