Google’s John Mueller Discusses Site Discovery Process
Google webmaster trends analyst John Mueller responded to a question on Reddit about how the search engine discovers sites without backlinks. The question was, “How does Googlebot find a site if no one is linking to the site, and it’s not been submitted to Search Console?” Mueller’s response went into some detail and offered some examples for how Google’s crawlers might still be able to discover a URL: “Sometimes it’s 3rd parties that track domain registrations (with links), sometimes it’s links with typos out there, sometimes it’s toolbars that people use (again, that link to related content), sometimes it’s just your CMS being helpful (say with a sitemap or RSS/Atom feed).” Mueller added that the best way to prevent a site from being found is to use the noindex tag, as this is the only sure way to prevent site indexing. He added that one way to launch a site “with a bang” would be to use Google’s site removal tool in Search Console, then cancel the request when it’s time to go live. “That lets Google crawl and index the content ahead of time, but prevents it from being shown in search.” According to Mueller, this is also a faster process than switching from noindex to indexable content, but other search engines and users might see it still. The site removal tool takes less than a day to take effect, and isn’t actually a full removal. According to Mueller, it’s “basically a ‘URL temporarily hide in the search results tool’” and doesn’t affect crawling and indexing. If you’re looking to keep a site out of sight until an official launch, noindex tags might be your best bet. But if you’re looking to make a splash when you launch, consider using the site removal tool to get your content indexed ahead of time.