Meet our Mobile Bots

In today’s post we are joined by Lee Xiong from the Bing Crawl team. Lee is going to discuss some new developments on the crawl front pertaining to mobile SEO.  Enjoy!  – Vincent

We can all agree that mobile is the future. Actually, we can’t really say “mobile is the future” anymore. Mobile is the present. Mobile is now. With that in mind, it’s time to take a fresh look at some essential things from the ground up. Specifically, let’s re-examine all the work you’ve been doing to get crawled, selected, indexed, and ranked. This time though, let’s look at it through a mobile lens.

Mobile Crawl to Inform our Rankers

We’ve blogged about the importance of mobile before, but it’s never too late for a reminder. And what we want to discuss today is mainly how Bing is actively checking your website for “mobile compatibility” —  an important aspect of how we view your site when we serve our results on Bing or Bing-powered search across mobile devices.

The Power of One

Our original recommendation to webmasters still applies when it comes to mobile: avoid duplication and prevent bifurcation of your ranking power by avoiding separate m.-URLs for mobile. Instead, move to responsive design that adapts to the device and benefit from maximum SEO power instilled in a single URL. This continues to be the way forward for future-looking sites.

Probing the Web for Mobile Friendliness

At the same time, we are cognizant of the fact that many sites still use different URLs for their mobile phone or smart phone customers or have varying levels of user experiences depending on the type of device. So, as true advocates for our users, we are very interested in understanding how your content “renders” on these devices and if it makes a for a good user experience. To that end, we have started to probe websites with a number of new crawlers with the aim to give us the best representation of what our users can expect from your website when viewed on their favorite device.  

Introducing our Bingbot Mobile User Agents

As you may know from our help topic Which Crawlers Does Bing Use, we have a number of crawlers to perform our common crawl duties. To understand how your site behaves specifically for our mobile searchers, we have added a couple of new crawler variants which identify themselves with a user agent that mimics some of the most common mobile device types. In general, these crawlers use a user agent string that follow the following format:

Mozilla/5.0 + (Mobile Device) + Mobile Engine + Mobile Browser + bingbot/BingPreview/[version]

Here are a couple of example user agent strings that follow this format:

  • Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b
  • Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko BingPreview/1.0b
  • Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
  • Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

In all of these examples, the user agent strings containing “BingPreview” refer to crawlers that are capable of “rendering” the page, just like a user’s browser would. It is therefore paramount that you allow our crawlers to not only find the core content of the URLs themselves, but that you also allow them access to the necessary resources needed to load each page, that is, including any CSS, script, and image files.

Conclusion

It’s no secret that mobile is pervasive and that performing well on mobile devices is becoming more and more important for websites to succeed. To safeguard the best mobile search experience for your users, move towards a responsive model that adapts to your user’s device at the same URL. At the same time, to safeguard the best mobile search experience for our mobile Bing users, we do probe the web with crawlers that emulate the most popular user devices (and which identify themselves as such!) to inform our rankers.  To that end, make sure our mobile bots can crawl your site freely and that you are not blocking essential parts of your site (such as JavaScript or CSS files).

Mobile on!

Lee Xiong –

Program Manager, Bing Crawl Team