AI Crawler Checker • Specialized Workflow

ClaudeBot & PerplexityBot Access & robots.txt Checker

Anthropic (ClaudeBot, Claude-SearchBot, Claude-User) and Perplexity (PerplexityBot, Perplexity-User) strictly honor robots.txt and X-Robots-Tag directives. Test your live domain in one click.

Technical Specification & Formula

Audited Agents: ClaudeBot • Claude-SearchBot • Claude-User • PerplexityBot • Perplexity-User • Google-Extended • Applebot-Extended

Step-by-Step: ClaudeBot & PerplexityBot Checker

  1. Paste Target URL. Enter the exact URL you want Anthropic and Perplexity crawlers to index and cite.
  2. Inspect Rule Precedence & Headers. Detect conflicting wildcard rules, meta noindex tags, or X-Robots-Tag noai directives.
  3. Verify HTTP 200 Reachability. Confirm your server responds cleanly without redirect loops or 403 blocks.

Frequently Asked Questions

Which user-agent controls Claude web search citations?
Anthropic uses Claude-SearchBot for search indexing, Claude-User for user-triggered fetches, and ClaudeBot for broader corpus crawling. Allowing Claude-SearchBot and Claude-User ensures Claude can cite your live pages.
Does Cloudflare Bot Fight Mode block PerplexityBot and ClaudeBot?
Yes, aggressive "Block AI Scrapers" toggles in Cloudflare or WAFs frequently return HTTP 403 to PerplexityBot and ClaudeBot even when robots.txt says Allow: /.

More AI Crawler Checker Workflows

  • Full AI Crawler Checker Workspace — Audit your website's robots.txt directives and X-Robots-Tag headers for AI crawlers, search bots, and training spiders.
  • GPTBot & OAI-SearchBot Checker — Test if GPTBot, OAI-SearchBot, and ChatGPT-User are allowed or blocked in your robots.txt and HTTP X-Robots-Tag headers. Free instant OpenAI crawler audit.

Related Free SEO Tools

  • LLMs.txt Generator & Validator — Create and validate official llms.txt and llms-full.txt files according to the latest specification for AI agents, developer tooling, and LLM search indexing.
  • AI Visibility Checker — Check whether ChatGPT, Gemini and Perplexity cite your website, mention your brand, or recommend competitors instead. Free, no sign-up, results in about a minute.
  • AI Link Hallucination Checker — Detect hallucinated and broken URLs in AI-generated drafts. Automatically verifies HTTP status codes, DNS validity, and SSL certificates.

GrowthControl — Track AI Visibility in ChatGPT, Gemini, Perplexity & Claude →

Audit whether AI search engines mention and cite your brand with our free AI Brand Visibility Checker.