AI Crawler

The short answer

An AI crawler is an automated bot — like GPTBot, PerplexityBot, ClaudeBot, or OAI-SearchBot — that visits web pages to gather content either for AI model training or for live search retrieval.

Full explanation

Blocking these bots in robots.txt removes your site from that platform's AI answers entirely, whether or not that was the intent. It's worth distinguishing training crawlers (which build future model knowledge) from search-retrieval crawlers (which power live, cited answers) before deciding what to block.

Back to all terms

77 terms and growing