In order to streamline numerous activities and lessen the workload on Googlebot, Google has launched a new web crawler called GoogleOther. Googlebot is the primary crawler used by Google. In essence, GoogleOther’s main responsibility is to conduct optional jobs like R&D crawls. As a result, the Googlebot will now concentrate mostly on expanding the search index.
The new crawler should also aid Google in streamlining and improving its crawling processes. For the uninitiated, Web crawlers—also referred to as robots or spiders—serve the purpose of automatically finding and scanning webpages. Different product teams can use GoogleOther to obtain publicly accessible content from different websites.
According to Google Search Analyst Gary Illyes on LinkedIn, several Google teams will use GoogleOther internally to scan the public internet.
It will free up some resources for the primary Googlebot crawlers while adhering to the same rules and regulations.
GoogleOther was added to its list of crawlers:
“Generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development. The GoogleOther crawler always obeys robots.txt rules for its user agent token and the global user agent (*), and uses the same IP ranges as Googlebot.”
Why the new bot?
According to Illyes, this new bot “ultimately will take some strain off of Googlebot”.
He also said that
“As we optimize how and what Googlebot crawls, one thing we wanted to ensure is that Googlebot’s crawl jobs are only used internally for building the index that’s used by Search. For this, we added a new crawler, GoogleOther, that will replace some of Googlebot’s other jobs like R&D crawls to free up some crawl capacity for Googlebot.”
It is utilized internally by Google. Does that imply that Bard will be using it? It’s not clear.
Should GoogleOther be blocked? What would happen if you attempted to block this new bot is unclear. However, I think you should keep an eye on it, and if it is using your server too much, you can choose whether to grant or prohibit access to your website to this new Googlebot.
Why it matters to us? Many of you monitor your website’s crawling activity, bot activity, and log files. Do not panic if you notice this new GoogleOther crawler. It’s a genuine Googlebot.
The introduction of this new bot may not have a significant impact on the websites. But website owners can ask for the help of SEO companies to keep an eye on website ranking. So they can immediately change the strategies accordingly if the bot influences the search engine ranking.
Related Links:
https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers