GoogleOther: Google’s New Web Crawler Designed to Optimize Crawling Operations
Google has unveiled its new web crawler called GoogleOther, which is designed to alleviate strain on Googlebot, the search index crawler. This addition will ultimately help streamline and optimize Google’s crawling operations. Web crawlers, also known as robots or spiders, automatically discover and scan websites, and Googlebot has been responsible for building the index for Google Search. The new GoogleOther, a generic web crawler, will be utilized by various product teams within Google to fetch publicly accessible content from websites.
Dividing Responsibilities Between Googlebot & GoogleOther
The primary purpose of GoogleOther is to take over the non-essential tasks currently performed by Googlebot. By doing so, Googlebot can focus solely on building the search index, while GoogleOther handles non-search indexing jobs such as research and development (R&D) crawls. This will free up some crawl capacity for Googlebot. Google Search Analyst Gary Illyes shared these details on a LinkedIn post.
GoogleOther Inherits Googlebot’s Infrastructure
GoogleOther shares the same infrastructure as Googlebot, which means it possesses the same limitations and features, including host load limitations, robots.txt, HTTP protocol version, and fetch size. Essentially, it is Googlebot operating under a different name.
Implications for SEOs & Site Owners
The introduction of GoogleOther should not significantly affect websites, as it operates within the same infrastructure and limitations as Googlebot. Nevertheless, it is a significant development in Google’s ongoing efforts to optimize and streamline its web crawling processes.
If you’re concerned about GoogleOther, here are some ways you can monitor it:
Analyze server logs
Update robots.txt
Monitor crawl stats in Google Search Console
Track website performance
Monitoring your website performance metrics such as load times, bounce rates, and user engagement will help detect if there are any unforeseen issues caused by the new crawler. Visit SEO Augusta’s SEO Services for more information.
Source: Google
Featured Image: BestForBest/Shutterstock
