New work details of the updated Googlebot have become known

2 minutes read

The other day, Google shared new details of the updated Googlebot, which will now always be based on the latest version of Chromium.

Old user agent

As it turned out, Google did not change the user agent for Googlebot and still uses the old version that mentions Chrome 41. In fact, the crawler is powered by Chrome 74, the company said.

At the same time, Google plans to update the user-agent in the future:

Tweets by g33konaut

Two workarounds for javascript pages

Google still processes JavaScript content in two rounds. Therefore, delays in its indexing remained, but support for more than 1000 new functions appeared.

Tweets by g33konaut

When asked whether there are plans to combine these two phases of indexing JS content into one, Google Split’s employee Martin Split responded with the standard phrase: “[We] have nothing to announce yet”.

Tweets by g33konaut

Scan volume will remain the same

When asked if the scanning volume will change – i.e., whether Googlebot will scan less pages with the same crawl budget due to processing more complex files – Split replied no.

Tweets by g33konaut

New features support

As previously reported , as a result of the update, Googlebot received support for over 1000 new features. Below are some of them.

Our session on #javascript and #seo at 1:30pm tomorrow will be LIT🔥🔥🔥
Come to stage 3 to see what it's all about 🙃

Amongst other things, we'll talk:
🏗️Web app architectures
⚡️Lazy loading
🚀Web APIs
📈Crawl budget & more!

— Martin Splitt @ 🇨🇭🏡 (@g33konaut) May 9, 2019

Googlebot has not yet been updated in all Google tools .

Googlebot is used in different services and divisions of Google, but not all of them have already updated the crawler. While the update touched only search. At the same time, the Mobile Optimization Test tool and other services have not yet switched to the updated version of Googlebot, but will be transferred to it in the future.

As noted research officer Gary Ilsh , update all at once Google does not have the possibility, this infrastructure is arranged differently. Therefore, the company will need some time to update all of its services.

New user- agent

Some webmasters also noticed in their log files the appearance of a new user agent that contains the following data: Android 9.0.0 Pixel 3 XL, 01-may-2019 .

Tweets by Errioxa

Tweets by glenngabe

Google has not yet answered what the user-agent is:

Tweets by g33konaut

Recall that Google introduced an updated version of Googlebot on the first day of the conference for I / O developers 2019.


Facebook Twitter LinkedIn Telegram Pocket

Related Posts:

Google announced the launch of a new version of Googlebot. Now the web crawler will always work based on the latest version of Chromium, the open source browser that Chrome was built on. At the moment, Googlebot uses Chromium version 74 instead of the long-obs...
Recently it turned out that the launch of the new Googlebot, which will now always work based on the latest version of Chromium, was launched several months ago. In November, the new crawler indexed 10% of the Internet, and last week this figure was increased ...
5 months ago Distilled agency employee Tom Anthony discovered a vulnerability that allows Googlebot to be manipulated to execute JavaScript and index its changes, including links. The researcher notified Google of his discovery, but the company did not close t...