// TOOL 08 — CRAWLER CONFIGURATION
llms.txt Generator
Draft a compliant llms.txt for your site so AI crawlers know what you are, what you do, and where your most important endpoints live.
// CONFIGURE
// OUTPUT
Why you need an llms.txt file
As AI search engines like Perplexity, SearchGPT, and Google AI Overviews replace traditional blue links, your visibility depends on how easily Large Language Models can parse and summarize your site.
While robots.txt tells bots what to ignore, llms.txt tells them what matters. Providing this file gives you direct control over how your entity is defined in the training data and live retrieval context of the world's most powerful AI models.
FAQ
What is an llms.txt file?
llms.txt is a standard proposed to help AI crawlers and Large Language Models understand the purpose, structure, and important endpoints of a website. It acts similarly to robots.txt, but instead of telling bots what NOT to crawl, it tells them what is MOST important to read.
Where do I put this file?
You should place the llms.txt file at the root of your domain (e.g., https://yourdomain.com/llms.txt). It is becoming standard practice for AI agents to look for this file when profiling a site.
Why should I care about AI crawlers?
Search is shifting from traditional index retrieval to AI-generated answers. If ChatGPT, Perplexity, or Google's AI overviews can't easily understand what your business does and where your docs/pricing are, they will exclude you from their generative answers.