Websites
Robots.txt
The short answer
Robots.txt is a small text file at the root of a website that tells crawlers — search engines and AI bots alike — which parts of the site they're allowed or not allowed to visit.
Full explanation
A misconfigured robots.txt is one of the most common, and most damaging, technical mistakes a site can make — accidentally blocking Googlebot, or an AI crawler like GPTBot or PerplexityBot, removes the entire site from that platform's results without any warning.
Back to all terms
77 terms and growing