Since our last post in November, the Bing team has been busy rolling out improvements to the Bing web crawler. As a result of this work, we want to announce in advance our plans to change the name of our crawler (aka user agent).
Out of beta with a new name
On October 1st, 2010, we will drop the beta designation from the Bing crawler and change the name of the crawler to reflect Microsoft’s new brand for search. Instead of the old msnbot 2.0b showing up in your server logs, the updated user agent will be:
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
The HTTP header From field will also change as shown below:
From: msnbot(at)microsoft.com
will become
From: bingbot(at)microsoft.com
Robots Exclusion Protocol directives precedence
We want webmasters to know that bingbot will still honor robots.txt directives written for msnbot, so no change is required to your robots.txt file(s).
Please note, however, that if we detect separate sets of directives for bingbot and for any of the older versions of Microsoft search bots (such as msnbot) or a set of directives for all crawlers, the directives for bingbot will take precedence. For example, in the following case, Bing will be authorized to crawl on all whole hosts except on the folder /folder1/, despite the more comprehensive blocking directives for other crawlers:
User-agent: bingbot
Disallow: /folder1/
User-agent: msnbot
Disallow: /folder1/
Disallow: /folder2/
User-agent: *
Disallow: /
How to report issues
We always welcome feedback from the webmaster community. Over the past few months, during the recent development phase, we had different issues reported to us. We want to give our heartfelt thanks to the webmasters who contacted us, and offer our sincere apologies for any issues they may have encountered as a result of the crawler traffic.
If you have any questions regarding BingBot traffic and your site, please contact us at bingbot@microsoft.com. This is the most effective way to reach us. We do our best to respond to your inquiries promptly and report back to you on issues brought to our attention. When you do so, please provide us with a snippet of your logs showing the BingBot traffic as well as a list of the sites served from the same infrastructure, if there is more than one. This way, we can more quickly diagnose and resolve the issues in question.
We look forward to seeing your comments and feedback. Thanks!
Additional bot-related postings:
- http://www.bing.com/community/blogs/webmaster/archive/2009/11/05/robots-speaking-many-languages.aspx
- http://www.bing.com/community/blogs/webmaster/archive/2009/11/04/msnbot-1-1-is-retired.aspx
- http://www.bing.com/community/blogs/webmaster/archive/2009/08/21/prevent-a-bot-from-getting-lost-in-space-sem-101.aspx
— Dean Wierman and Fabrice Canel, Bing Crawl and Index teams