Bingbot is coming to town

Back in June, 2010, we published a blog post titled Bing crawler: bingbot on the horizon that announced our plans to retire our venerable web crawler, MSNBot, and replace it with the new. Our plans remain on track, and we want to remind you that this change will occur on October 1st, 2010.

We also want to take this opportunity to help set some expectations for this process, to discuss the name change details, what the change means in terms of control crawler access to your content, and finally cover what is not changing as part of this process.

Name change

As of October 1st, you’ll start seeing the following user agent name in your server logs:

Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

The old HTTP header From field, currently seen as:

From: msnbot(at)microsoft.com

will be renamed to the following:

From: bingbot(at)microsoft.com

On October 1st, you should see the main Bing web crawler name change implemented. Other MSNBot crawlers will be phased out shortly thereafter.

The renamed crawler will continue to perform as it did previously. You should not see any differences stemming from this change.

Bot access and control

It is critically important that you review your server configurations now for crawler access. Webmaster-managed bot access control methodologies, such as robots.txt files, robots <meta> tag directives, and HTTP headers, may need to be adjusted if you normally block all crawlers except for specific, preferred search crawlers.

Note that we’ll still honor all applicable robots.txt directives, including those specific to MSNBot. That said, as of October 1st, if we detect separate sets of custom directives for both MSNBot and BingBot in your robots.txt file, the BingBot directives will take precedence.

If your sites are configured for any special handling based on user agent (such as via the .htaccess file), you’ll need to update this for BingBot so we can continue to crawl your sites as before.

To keep your content in the Bing index, please ensure that BingBot has the same access to your site as given to MSNBot. Given the recent developments with Microsoft and Yahoo! Search Alliance, where Bing is now also powering Yahoo! search results for web, image, and video content, this issue is all the more important.

Note: If you previously blocked the Bing crawler but allowed the Yahoo! crawler access to your site, you’ll now need to allow access to BingBot to continue to be seen in Yahoo! search results.

What’s not changing

The IP addresses used by our crawlers will remain the same. The crawl rate will not be affected by the name change. And, of course, the best way to get BingBot to crawl and index more of your site is to use standard search engine optimization (SEO) techniques. For more information, see our series of posts on performing SEO site reviews, starting with Is your site ranking rank? Do a site review – Part 1 (SEM 101).

We also highly recommend registering your sites with the new Bing Webmaster Tools, where you can add or block URLs, submit Sitemaps, check for crawler-detected problems, get tons of information about your site, and much more.

If you have any questions, comments, or suggestions, feel free to post them in our Indexing and Ranking Discussion forum. Later…

– Rick DeJarnette, Bing Webmaster Center

Join the conversation

33 comments
  1. yasirwazir

    Thank You for this update

  2. Quality Directory

    This blog post is more detailed than the one published in June, and has cleared some of my doubts.

  3. DualRack

    Thanks for this information!!

  4. Medical Stubs

    thanks..

  5. calderk

    Good news.

  6. lilia_dayspa

    just found this I been working on seo for google. I think bing is better. thanks for the info. I trying to get my site submitted to bing now.

    dyuane

  7. coachhandbags123

    like this blogs

  8. cellhub

    Thanks Rick, for sharing this information about the Bing BOT.

  9. royhunters

    How about fixing the site verification problem in webmaster tools

  10. ballabhyadav

    awesome

    google is out now !!

  11. bouka55

    Good, I hope its also more frequent a visitor than its predecessor!

  12. lheitts82

    With this new progress, Bing and Yahoo could surely get even with Google. It is really great to have our websites ranked well in this three search engines

  13. casp_40

    Looking forward to the BingBot crawling my websites!

  14. jongarci

    Cool

  15. mynewmachine.com

    Cool :)

  16. jollytrade01

    Thank you!

  17. syntax

    Thanks.

    This is the bing's food voices :)

  18. Anonymous

    so ! is this neccessary to add to the bot list ???

  19. blogtactic

    noted

  20. Toys And Games

    Definitely good to know. Go get em bing bot.

  21. Toys And Games

    Good to know.

  22. eLinenStore.com

    Re: It is critically important that you review your server configurations now for crawler access. Webmaster-managed bot access control methodologies, such as robots.txt files, robots <meta> tag directives, and HTTP headers, may need to be adjusted if you normally block all crawlers except for specific, preferred search crawlers.

    If there are no specific mods to the crawler access currently, do we need to add anything specific to ensure that access is granted?

    Thank you,

    John

  23. timtitolo

    Waiting to see how this all works

  24. fredericodejong

    Great, now google will have to come off the pink cloud.

  25. JSherrod

    Thanks for the update and timing information!

  26. coolit_jackson

    Thanks..

  27. Roobs

    Good to know, thx for the update. Even though I don't have much crawlable content myself, it's good to have the info if someone asks.

  28. tripvera

    Thanks for the update

  29. shawneetee

    It seems that the last time MSNBOT visited our sites on October 12, 2010, it removed all of our sites from bing.com and Yahoo.com. bingbot has not visited our sites yet so we have disappeared.

    We did not make a single change to our sites in months so the only reason we can find is that MSNBOT had an issue on the last crawl. When will bingbot visit our sites to restore us to the index?

  30. sohel

    Woah this is just an insane amount of information, must of taken ages to compile so thank you so much for just sharing it with all of us. If your ever in any need of  related info, perhaps a bit of  coaching, seduction techniques or just general  tips, just check out my own site!

    http://bobevanscoupon.net

  31. www.cool4arab.net

    Why my website is not like working on bing? but it works fine with google?

    http://forum.fashel.net

    please help

Comments are closed.