Google describes it as a “generic crawler” used by various teams within the company.

This vagueness has sparked curiosity and concern among website owners. Is this bot a helpful assistant, or a potential privacy threat? Let’s break it down.

What does GoogleOther do?

The short answer is: we don’t really know.

Google isn’t particularly forthcoming about the bots exact role. Here is the official definition of GoogleOther:

“GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.”

We do know it can crawl regular web pages, images, and videos. Beyond that, its purpose remains shrouded in mystery.

Speculation runs rampant. Some believe it’s involved in research and development for new Google products. Others worry it might be collecting data for less benevolent purposes.

Should you block GoogleOther?

The decision to block GoogleOther isn’t straightforward. There are pros and cons to consider.

Potential reasons to block:

• Privacy concerns: If you’re concerned about Google collecting data from your website, you might consider blocking GoogleOther. However, it’s unclear if this action will actually protect your privacy..
• Resource management: While unlikely to be a major issue, blocking GoogleOther could save a small amount of server resources.

Potential reasons to allow :

• Potential benefits: GoogleOther might be contributing to the development of features that could benefit your website. Blocking it could prevent you from accessing these potential opportunities.
• Google’s recommendation: Google explicitly advises against blocking GoogleOther. Disobeying this advice might lead to unexpected issues with other Google services.

Recently, Gary Illyes from Google answered questions about this peculiar web crawler and its support for non-search features.

“What non-search features does GoogleOther crawling support?”

While Gary Illyes did not answer about the non-search features that GoogleOther crawling supports, he does recommend not to block it, as it can potentially cause problems.

“This is a very topical question, and I think it is a very good question. Besides what’s in the public I don’t have more to share.

GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.

Historically Googlebot was used for this, but that kind of makes things murky and less transparent, so we launched GoogleOther so you have better controls over what your site is crawled for.

That said GoogleOther is not tied to a single product, so opting out of GoogleOther crawling might affect a wide range of things across the Google universe; alas, not Search, search is only Googlebot.”

What’s the takeaway?

Unfortunately, Gary Illyes didn’t answer with as much clarity as we usually expect from him. But what did we learn?

We learned that this mystery bot may have an effect on a wide range of things, including other Google products and services. And we learned, from Gary, that we should not block it.

If you are concerned about privacy, we recommend taking other measures to ensure what you want to protect is secure. But until we know more, it is relatively safer to keep allowing GoogleOther to crawl your website than the other way around.

The post What does GoogleOther do? appeared first on Improve My Search Ranking.

A sitemap allows search engines to crawl your site more intelligently and work out which pages are most important.

It is also good practice to specify your sitemap’s location in your robots.txt file.

The post Sitemap appeared first on Improve My Search Ranking.

The post Mobile Responsiveness appeared first on Improve My Search Ranking.