Practical guide

AI search crawlers: ChatGPT, Claude, Perplexity and Google

Allowing a search crawler makes a public page a retrieval candidate. It does not guarantee indexing or recommendation. Check both website directives and actual network access.

Sivlo · Updated

OpenAI: search and training are separate choices

OAI-SearchBot supports OpenAI search. GPTBot concerns potential model training, and ChatGPT-User concerns user-initiated requests. Check the role of each in official documentation. Pursuing search visibility does not require allowing training.

Claude and Perplexity

Anthropic distinguishes Claude-SearchBot, Claude-User and the training-related ClaudeBot. Perplexity documents its own search crawlers and user-directed retrieval. Check current names and verification guidance directly with each provider; published IP information differs between services.

Google: indexing eligibility and useful content

Google’s generative search guidance builds on search technical requirements and useful content. Inspect page indexing and available search-feature reports in Search Console. Adding a URL to a sitemap does not confirm indexing.

A developer verification checklist

Check the actual HTTP status and final URL. Confirm the public page has no unintended noindex directive and that its content appears in readable HTML. Review robots.txt and CDN, firewall and bot-protection decisions. Verify a genuine crawler according to provider guidance: user-agent text alone is not identity proof.

Keep workspaces, customer reports and APIs behind authentication and authorization. Robots.txt directs crawler behavior; it does not protect private data. Do not expose application data for visibility.

Sources and further reading

Measure your own market

See who AI recommends in your market, why, and what to change.

Request a demo