Live Search is committed to continuing to improve how we discover and crawl your content. In the coming weeks, we will be testing an update to MSNBot, which may show up as a new crawler name in your referrer logs. The new crawler user agent string will appear as:
msnbot/2.0b (+http://search.msn.com/msnbot.htm)You’ll still see the current version msnbot/1.1 (+http://search.msn.com/msnbot.htm) as Live Search’s primary user agent. We intend to ensure that any robots exclusion protocol you are using is respected. As such, you don’t need to update your Robots.txt file. We plan on crawling at a slow speed during the tests with the updated version.
When the new crawler replaces our current crawler, we will keep you up-to-date here on the Webmaster Center blog. If you have any questions about the Live Search web crawlers, you can post them in our forums.
Additional posts you should read:
- Robots Exclusion Protocol: Joining Together to Provide Better Documentation
- Announcing Crawler Improvements for Live Search
- Search robots in disguise
Jeremiah Andrick - Program Manager - Live Search Webmaster Center