✨ Say goodbye to lost search rankings. Introducing Gobiya AI Citation Engine. Learn more
Technical SEO and indexing decide whether a page is eligible to rank at all. A page that is never crawled is never indexed, and a page that is never indexed cannot rank for anything, however good the writing is. The eleven terms below are the checks that determine eligibility, roughly in the order a crawler encounters them.
A page no crawler reaches cannot fail later checks. It never gets to them.
Crawl budget is the limited number of pages a search engine's bot will crawl on your site within a given period of time.
On large sites, wasting crawl budget on low-value pages (broken links, duplicate content, endless filter combinations) can mean your genuinely important pages get crawled less often — or not at all.
Robots.txt is a small text file at the root of a website that tells crawlers — search engines and AI bots alike — which parts of the site they're allowed or not allowed to visit.
A misconfigured robots.txt is one of the most common, and most damaging, technical mistakes a site can make — accidentally blocking Googlebot, or an AI crawler like GPTBot or PerplexityBot, removes the entire site from that platform's results without any warning.
An XML sitemap is a file that lists every important URL on your site, submitted to search engines to help them find and understand your pages more efficiently.
A sitemap doesn't guarantee indexing, but it removes the guesswork for a crawler, especially on large sites or ones where internal linking alone might not surface every page quickly.
Being crawled is not being stored. These decide whether it is kept.
Indexation is the process of a search engine storing a crawled page in its index, making it eligible to appear in search results at all.
A page can be perfectly written and still invisible in search if it was never indexed — common causes include being blocked in robots.txt, marked "noindex," or simply never discovered because nothing links to it.
A canonical URL is the one "official" version of a page that a site tells search engines to index, when the same or very similar content is reachable at more than one web address.
Without a canonical tag, duplicate or near-duplicate pages (like a product page reachable through two different category paths) can split ranking signals or cause the wrong version to get indexed.
Mobile-first indexing means Google primarily uses the mobile version of your website — not the desktop version — to decide how to rank and index your pages.
If content, links, or structured data are missing from the mobile version of a page, Google may effectively treat them as missing altogether, even if they're present on desktop.
The basic hygiene checks that cost rankings quietly when they fail.
A broken link is a link that leads to a page that no longer exists, usually showing a "404 Not Found" error instead of real content.
Broken links waste crawl budget, frustrate visitors, and — if enough of them build up — can quietly erode a site's perceived quality in Google's eyes. Redirecting removed pages to a relevant live page instead of leaving a 404 preserves both the visitor experience and any inbound links pointing at the old URL.
An SSL certificate is what encrypts the connection between a visitor's browser and your website, shown in the browser as the padlock icon and a URL starting with "https://" instead of "http://".
Google has confirmed HTTPS as a ranking signal, and browsers actively flag non-HTTPS sites as "Not Secure" — both a trust problem for visitors and a quiet SEO penalty for going without one.
Decisions made once that constrain everything after them.
A domain name is the address people type to reach your website, like "gobiya.com" — it's also the identity search engines and AI tools use to track your site's reputation over time.
A domain's age, history, and consistency all factor into how much trust engines place in it — which is one reason changing domains carelessly can cause a temporary, sometimes lasting, drop in visibility.
A CMS is the software a website is built and edited on — like WordPress, Webflow, or a custom system — letting people update content without writing code directly.
Different CMS platforms handle SEO fundamentals like clean URLs, page speed, and schema markup very differently out of the box, which is why the same content can perform very differently depending on the platform it's published through.
The one tool that reports what Google actually did, rather than what should have happened.
Google Search Console is a free tool from Google that shows how your site actually performs in search — impressions, clicks, average position, indexing issues, and more.
It's the primary source of truth for diagnosing SEO problems: whether pages are indexed, which queries drive traffic, and where impressions are climbing while clicks stay flat — often an early sign an AI Overview is now answering the query directly.
Use the URL Inspection tool in Search Console before changing anything — it reports the specific reason. The common causes are a noindex tag left in place, a canonical pointing elsewhere, a robots.txt rule blocking the path, or a page Google crawled and judged too thin to store. Each has a different fix and guessing wastes weeks.
Rarely. Crawl budget becomes a real constraint in the tens of thousands of URLs, or where faceted navigation generates near-infinite combinations. A few hundred pages will be crawled comfortably, and time spent on crawl budget there is time not spent on something that matters.
It is not strictly required, but it is close to free and it gives Google a definitive list of the URLs you consider canonical. It matters most for new sites with few inbound links, and for large sites where internal linking alone leaves pages several clicks deep.
It is a lightweight one, and that undersells the point. Browsers now mark plain HTTP pages as insecure, so the cost of not having a certificate is paid in visitor trust long before it shows up in rankings.