How Often Does Google Crawl and Index A URL?

author image Written by: Rob May           Categories - Digital Marketing

Webmaster Hangout Provides Insight into Frequency

During a recent webmaster hangout, Google’s John Mueller provided answers about how often the search engine crawls URLs. Answering a question about why a page set to noindex was still showing up in Google’s index, Mueller offered some insight into Google’s approach to crawling updated pages and what webmasters and SEOs could do on their end. “ don’t crawl URLs with the same frequency all the time,” said Mueller. “So some URLs we will crawl daily. Some URLS maybe weekly. Other URLs every couple of months, maybe even once a half year or so.” Mueller stressed that Google continually tries to find the right balance for this, “so that we don’t overload your server.”

What Triggers a Crawl?

Mueller offered some key insights into the crawl process and what webmasters can do to help trigger it. Without providing too much detail about Google’s processes, he did explain that big updates to a site were often a clear trigger for a crawl. “If you made significant changes on your website across the board, then probably a lot of those changes are picked up fairly quickly,” he said, “but there will be some leftover ones.” Circling back to the question that sparked the answer, he added, “If you do things like site queries then there’s a chance that you’ll see those URLs that get crawled like once every half year. They’ll still be here after a couple of months.” He did suggest updating sitemaps and allowing Googlebot to find the last modified date could lead to a crawl of the pages in question.

Other Options

Of course, there are other ways to get Google to crawl a website. Using Google’s URL Inspection Tool, users can actually manually request indexing or reindexing. It’s a simple process that can be started from the Tool’s reports, but it will take up to two weeks for your request to make it through the queue and the indexing process. This is handy if you just need to reindex a page or two. Want to learn more about how Google handles re-crawl and reindexing requests? Check out the official process here: https://support.google.com/webmasters/answer/6065812 Watch the full Webmaster Hangout here: https://www.youtube.com/watch?v=gC7aVygbMIk&feature=

Rob May

Google Goes Mobile-First When Indexing New Sites

May 29, 2019

Mobile-First Indexing Enabled by Default for All Sites Starting July 1, 2019 We all knew this was coming: after years of talk about Google’s index prioritizing mobile-friendly sites, it’s finally happening. As of July 1, 2019, Google’s indexing will be mobile-first for all new websites and domains. According to Google, this change applies to new

read more

Bug in Google Search Causing Issues with Canonical URLs and Indexing

April 26, 2019

April A Bad Month for Indexing Issues as Google Continues to Work on Fixes and Solutions Late in the day on April 25, Google announced they’d found a bug in Google Search that could cause the search engine to select unrelated canonical URLs. According to Google’s Twitter announcement, this issue could also prevent proper indexing

read more

Leave a Reply

Your email address will not be published. Required fields are marked *