How Often Does Google Crawl and Index A URL?

author image Written by: Rob May           Categories - Digital Marketing

Webmaster Hangout Provides Insight into Frequency

During a recent webmaster hangout, Google’s John Mueller provided answers about how often the search engine crawls URLs. Answering a question about why a page set to noindex was still showing up in Google’s index, Mueller offered some insight into Google’s approach to crawling updated pages and what webmasters and SEOs could do on their end. “ don’t crawl URLs with the same frequency all the time,” said Mueller. “So some URLs we will crawl daily. Some URLS maybe weekly. Other URLs every couple of months, maybe even once a half year or so.” Mueller stressed that Google continually tries to find the right balance for this, “so that we don’t overload your server.”

What Triggers a Crawl?

Mueller offered some key insights into the crawl process and what webmasters can do to help trigger it. Without providing too much detail about Google’s processes, he did explain that big updates to a site were often a clear trigger for a crawl. “If you made significant changes on your website across the board, then probably a lot of those changes are picked up fairly quickly,” he said, “but there will be some leftover ones.” Circling back to the question that sparked the answer, he added, “If you do things like site queries then there’s a chance that you’ll see those URLs that get crawled like once every half year. They’ll still be here after a couple of months.” He did suggest updating sitemaps and allowing Googlebot to find the last modified date could lead to a crawl of the pages in question.

Other Options

Of course, there are other ways to get Google to crawl a website. Using Google’s URL Inspection Tool, users can actually manually request indexing or reindexing. It’s a simple process that can be started from the Tool’s reports, but it will take up to two weeks for your request to make it through the queue and the indexing process. This is handy if you just need to reindex a page or two. Want to learn more about how Google handles re-crawl and reindexing requests? Check out the official process here: https://support.google.com/webmasters/answer/6065812 Watch the full Webmaster Hangout here: https://www.youtube.com/watch?v=gC7aVygbMIk&feature=

Rob May

Google Goes Mobile-First When Indexing New Sites


Mobile-First Indexing Enabled by Default for All Sites Starting July 1, 2019

We all knew this was coming: after years of talk about Google’s index prioritizing mobile-friendly sites, it’s finally happening. As of July 1, 2019, Google’s indexing will be mobile-first for all new websites and domains. According to Google, this change applies to new websites, not existing pages, so the good news is you don’t need to drastically overhaul your site for this change. Google will watch and evaluate pages for mobile-first indexing as they have in the past for older sites. There’s more good news: dynamic serving and separate URLs for mobile sites will still be supported. Google does recommend responsive web design for new sites, however. From Google’s official Webmaster Blog announcement: “For existing websites we determine their readiness for mobile-first indexing based on parity of content (including text, images, videos, links), structured data, and other meta-data (for example, titles and descriptions, robots meta tags). We recommend double-checking these factors when a website is launched or significantly redesigned.” Considering how ubiquitous mobile searches are, it’s a bit surprising it’s taken this long for Google to go completely mobile-first. Google’s comments about this announcement do shed a bit of light on their reasoning. “Our analysis has shown that new websites are generally ready for this method of crawling.” It could simply be that they felt they gave ample notice about this over the years and are ready to make it happen. Considering the amount of talk and discussion mobile-first has gotten in the SEO community as far back as 2017, it looks like most webmasters have had time to prepare for this. Is your website built from the ground-up to be mobile-friendly, first and foremost? If you’re planning a redesign between now and July, make sure you get a responsive site that prioritizes mobile UX!

read more

Bug in Google Search Causing Issues with Canonical URLs and Indexing


April A Bad Month for Indexing Issues as Google Continues to Work on Fixes and Solutions

Late in the day on April 25, Google announced they’d found a bug in Google Search that could cause the search engine to select unrelated canonical URLs. According to Google’s Twitter announcement, this issue could also prevent proper indexing in some cases: https://twitter.com/googlewmc/status/1121538876126969859 Google also commented that this could also cause new indexing issues in cases where the search engine won’t index specific URLs. As of April 26, this issue still hasn’t been officially resolved, though Google is working on finding a solution. This issue comes on the heels of a Google News indexing problem just before Easter. This bug caused news content from major publishers to simply not index properly. And a week before that, Google was confirming they’d dealt with another indexing issue. Basically, April hasn’t been a good month for Google’s indexing abilities. Pages that aren’t in Google’s search index are, essentially, invisible to the average user through organic search. Unless someone has the exact URL, unindexed or deindexed pages won’t turn up in Google’s search engine results pages. Hopefully, Google will find a timely solution to these issues, and the SEO community can breathe a collective sigh of relief as things get back to normal.

read more

Leave a Reply

Your email address will not be published. Required fields are marked *