Back in June, 2010, we published a blog post titled Bing crawler: bingbot on the horizon that announced our plans to retire our venerable web crawler, MSNBot, and replace it with the new. Our plans remain on track, and we want to remind you that this change will occur on October 1st, 2010.
We also want to take this opportunity to help set some expectations for this process, to discuss the name change details, what the change means in terms of control crawler access to your content, and finally cover what is not changing as part of this process.
Name change
As of October 1st, you’ll start seeing the following user agent name in your server logs:
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
The old HTTP header From field, currently seen as:
From: msnbot(at)microsoft.com
will be renamed to the following:
From: bingbot(at)microsoft.com
On October 1st, you should see the main Bing web crawler name change implemented. Other MSNBot crawlers will be phased out shortly thereafter.
The renamed crawler will continue to perform as it did previously. You should not see any differences stemming from this change.
Bot access and control
It is critically important that you review your server configurations now for crawler access. Webmaster-managed bot access control methodologies, such as robots.txt files, robots <meta> tag directives, and HTTP headers, may need to be adjusted if you normally block all crawlers except for specific, preferred search crawlers.
Note that we’ll still honor all applicable robots.txt directives, including those specific to MSNBot. That said, as of October 1st, if we detect separate sets of custom directives for both MSNBot and BingBot in your robots.txt file, the BingBot directives will take precedence.
If your sites are configured for any special handling based on user agent (such as via the .htaccess file), you’ll need to update this for BingBot so we can continue to crawl your sites as before.
To keep your content in the Bing index, please ensure that BingBot has the same access to your site as given to MSNBot. Given the recent developments with Microsoft and Yahoo! Search Alliance, where Bing is now also powering Yahoo! search results for web, image, and video content, this issue is all the more important.
Note: If you previously blocked the Bing crawler but allowed the Yahoo! crawler access to your site, you’ll now need to allow access to BingBot to continue to be seen in Yahoo! search results.
What’s not changing
The IP addresses used by our crawlers will remain the same. The crawl rate will not be affected by the name change. And, of course, the best way to get BingBot to crawl and index more of your site is to use standard search engine optimization (SEO) techniques. For more information, see our series of posts on performing SEO site reviews, starting with Is your site ranking rank? Do a site review – Part 1 (SEM 101).
We also highly recommend registering your sites with the new Bing Webmaster Tools, where you can add or block URLs, submit Sitemaps, check for crawler-detected problems, get tons of information about your site, and much more.
If you have any questions, comments, or suggestions, feel free to post them in our Indexing and Ranking Discussion forum. Later…
— Rick DeJarnette, Bing Webmaster Center