// TOOL 05 — CRAWL & INDEX: ONLINE
Index & Crawl Analyzer
Before a page can rank, a search engine has to be able to crawl it and choose to index it. This analyzer inspects the machinery that governs both — your robots.txt and sitemaps — and flags the blocks and gaps that leave pages undiscovered or excluded.
What the analyzer checks
- C-01
robots.txt access
Whether robots.txt exists, whether it blocks crawlers site-wide (a catastrophic but common accident), and whether it declares your sitemap.
- C-02
Sitemap discovery
Fetches your XML sitemap — following sitemap-index files to sample child sitemaps — counts URLs, and checks for the lastmod dates that guide efficient recrawling.
- C-03
Scale & bloat signals
Flags sitemaps past the 50,000-URL limit and large sitemap indexes where index bloat and thin programmatic pages are the usual risk to crawl budget.
- C-04
Homepage indexability
Confirms your homepage isn't carrying a leftover noindex directive that would drop it — and often the whole site's authority — from search.
Related
// THE PROBLEMA thousand pages published, forty indexed — see the full diagnosis →// FLAGSHIPRun the full Website Health Scan on any single page →FAQ
What causes pages to not get indexed?
The common culprits: a robots.txt Disallow rule blocking the section, a noindex tag, missing or broken sitemaps so pages are never discovered, and — at scale — thin or duplicate programmatic content that Google crawls and declines to index ('Discovered — currently not indexed'). This tool checks the crawlability side; the content side is the other half.
What does this analyzer check?
It fetches your robots.txt and reads whether it blocks crawlers and declares your sitemap; fetches your XML sitemap (following sitemap-index files to sample child sitemaps); counts URLs; checks for lastmod dates; flags oversized sitemaps beyond the 50,000-URL limit; and confirms your homepage isn't accidentally set to noindex.
Does a bigger sitemap mean better SEO?
No — often the opposite. Pushing tens of thousands of thin, near-duplicate URLs into a sitemap invites index bloat, which dilutes crawl budget and can drag down your sitewide quality signal. Fewer, genuinely distinct, high-value pages usually index and rank better than a huge low-quality set.
Should my sitemap be listed in robots.txt?
Yes. A `Sitemap:` directive in robots.txt lets every search engine discover your sitemap automatically, even ones you never manually submit to in a webmaster console. It's a one-line change with a real discovery benefit.