Google’s Updated Bot Documentation: What You Need to Know

Recently, Google updated their Search Central Documentation to add information about user-triggered bot visits, which had been missing from previous Googlebot documentation. This led to confusion among many publishers, who ended up blocking the IP ranges of legitimate visits. In this article, we’ll go over the newly updated bot documentation and what it means for SEO.

The Three Categories of Google Bots

Google added documentation that categorizes the three different kinds of bots that publishers should expect. These are:

• Googlebot – Search crawler
• Special-case crawlers
• User-triggered fetchers (GoogleUserContent)

The last category, GoogleUserContent, has confused publishers for a long time because Google didn’t have any documentation about it. However, the new documentation explains that bot activity from IP addresses associated with GoogleUserContent.com can be triggered by the Google Site Verifier tool.

What You Should Know About GoogleUserContent

In the past, some in the SEO community believed that bot activity from IP addresses associated with GoogleUserContent.com was triggered when a user viewed a website through a translate function that used to be in the search results. However, Google’s new documentation does not confirm this. Instead, it only mentions the Google Site Verifier tool as a trigger. Additionally, Google does not say what else might trigger a bot from the GoogleUserContent.com IP addresses.

Another important update is the new reference to googleusercontent.com in the context of IP addresses that are assigned to the domain name, GoogleUserContent.com.

Google Bot Identification Documentation

The new documentation finally has something about bots that use IP addresses that are associated with GoogleUserContent. Search Marketers were confused by those IP addresses and assumed that those bots were spam.

Now we know that bot activity from IPs associated with GoogleUserContent are not spam or hacker bots. They are really from Google. Publishers who are currently blocking IP addresses associated with GoogleUserContent should probably unblock them.

To learn more about verifying Googlebot and other Google crawlers, check out Google’s updated documentation. For professional SEO services in Augusta, GA, visit SEO Augusta.

Featured image by Shutterstock/Asier Romero