Another crawler in your logs

Live Search is committed to continuing to improve how we discover and crawl your content. In the coming weeks, we will be testing an update to MSNBot, which may show up as a new crawler name in your referrer logs. The new crawler user agent string will appear as:

msnbot/2.0b (+http://search.msn.com/msnbot.htm)

You’ll still see the current version msnbot/1.1 (+http://search.msn.com/msnbot.htm) as Live Search’s primary user agent. We intend to ensure that any robots exclusion protocol you are using is respected. As such, you don’t need to update your Robots.txt file. We plan on crawling at a slow speed during the tests with the updated version.

When the new crawler replaces our current crawler, we will keep you up-to-date here on the Webmaster Center blog. If you have any questions about the Live Search web crawlers, you can post them in our forums.

Additional posts you should read:

Jeremiah Andrick – Program Manager, Live Search Webmaster Center

Join the conversation

33 comments
  1. Anonymous

    this means 2 msn crawler will be visiting my site is it? Was it already rolled out, or will just implement it?

  2. Anonymous

    The link to the forums is b0rken.

  3. Anonymous

    can’t find papaleonesspaghettihouse.com on live search why ? can on yahoo & google it is my site can you help me thanks DaddyRock@comcast.net

  4. Anonymous

    If you type in  papa leones spaghetti house  you will not fine my website just reviews of my restaurant. but you can on google or yahoo? Now  put in  papa leones spaghetti house.com  the page will be on top with porn. It does not belong with porn & stuff like that. You should type in    papa leones spaghetti house   press search and the website should appear with all of the reviews.  Why no one knows how to fix this. I have talk to everybody with MSN help thanks YOU CAN CALL 410-456-7278

  5. Anonymous

    good news,My bolog have been indexed 2 pages.waiting

  6. Anonymous

    my blog do not get indexed. How i can

  7. Anonymous

    If you type in papa leones spaghetti house you will not find my website just reviews of my restaurant  But put in papa leones spaghetti house.com the page will be on top.  It does not belong with porn & stuff like that. You should type in papa leones spaghetti house press search and the website should appear with all of the reviews.  Why no one knows I have talk to everybody with MSN help thanks  

    .

    type this in and it should be web page and reviews

    papa leones spaghetti house = just reviews

    type this in it should not be in with porn  and stuff

    papa leones spaghetti house.com= webpage & porn & stuff

    customers  cant find website please help call 410-456-7278

  8. Anonymous

    how long will it take you to come over here and fix it

  9. Anonymous

    if Another crawler in my logs show me so ican take it out !

  10. Anonymous

    You’ll still see the current version msnbot/1.1 (+http://search.msn.com/msnbot.htm) as Live Search’s primary user agent. We intend to ensure that any robots exclusion protocol you are using is respected. As such, you don’t need to update your Robots.txt file. We plan on crawling at a slow speed during the tests with the updated version what does that mean ?

  11. rickdej

    Srednarb,

    The new bot is just being tested. You may see it, you may not. We just wanted to make sure that if you saw it, you would know it was a Microsoft bot and not that of someone spoofing us.  

    Thanks

    Jeremiah

  12. rickdej

    daddyrock,

    This statement means that we are just testing this update to the bot and we are not rolling it out completely.  We don’t want to overcrawl or crawl to hard on your servers so we are limiting the amount of crawling this bot will do.

    Thanks

    Jeremiah

  13. Anonymous

    Will this crawler actually work unlike the others? Seriously, submitted a site 3 months back, it’s only indexed 3/30+ pages despite submitting sitemaps. Google took a couple of days to index the whole lot. No wonder noone uses Live, the content isn’t there because you fail to crawl. Clients won’t spend money on advertisering on Live if there websites aren’t even indexed. I think the Live team need to wake up.

  14. Anonymous

    Thanks for sharing this information blogger and SEO must aware..

  15. Anonymous

    thanx for nice artichle wery much msdn

  16. Anonymous

    I’m really curious about how my blog could be crawled by MSNBot. Should I modify meta tag or what?

  17. Anonymous

    To those with concerns about the number of pages of their site indexed let me say that identifying and removing duplicate content helped me a lot.  Just think about it.  If a bot is spending time crawling duplicate pages or content you don’t want indexed, that is taking away time it could be crawling the pages you do want indexed.  This is especially true for dynamic sites.

  18. Anonymous

    As of today, msnbot/2.0b  has not crawled my web site. Is their a code I can use to have msnbot/2.0b search all my pages when it does come by. All I have seen is how to stop it.

  19. Anonymous

    I’m yet to notice this new bot in my log statistics, except the usual MSNBot and MSNBot-media.

  20. Quality Directory

    I have never noticed any other crawler except the regular MSNBot.

  21. luke.sarker

    I agree with "Quality Directory", me too didn't find any other crewler except the regular MSNBot.

  22. aelusiveone

    The main thing is that you need quality incoming links to your site so that the crawler can follow those links and find your website. it worked well with my goji juice site and i never submitted the goji juice site directly to Bing

  23. Blackpool UK

    I have only noticed the msnbot too

  24. Anonymous

    How should I know the MSNbot is working with my site?

  25. miles2go

    Hope will work fast and better then previous one.

  26. amfamily77

    several time,i can do create my acount from Bing in my phone not PC and Computer. Just now,we can submit URL your site also,enter meta tag from webmaster.I optimize after add meta tag in your blog,with fast index your page in search engine like Bing.

  27. jamie_d

    Sorry idiots but I had to reject all ip addresses from your network because your bot has been asynchronously  traversing my website via multiple ip addresses and ignoring completely my robots.txt file,  which, in my opinion, translates to a DOS attack. Let me know when and if you decide to play by the rules or if you get your *** together, whichever comes first  …

  28. jamie_d

    I want you to know that I completely understand if you can't find good people to help you work this out since no one wants to work for you monkeys anymore …

Comments are closed.