More crawling improvements from MSNBot

A few months ago we announced two new features to MSNBot to reduce the burden of crawling on your website. These were part of a series of improvements we’re making to our crawler during the Spring to increase the freshness and breadth of content in our index. As part of these latest improvements, you may notice an increase in the amount of traffic from MSNBot starting over the next couple weeks. If you notice any issues with MSNBot, please make sure to drop us a note on our Crawling Feedback & Discussion Forum so we can investigate.

This is a great time to take a look at your robots.txt file (and meta tags) to make sure that you are not inadvertently blocking robots from content on your site you may want indexed. Also, if you feel that MSNBot is crawling your site too frequently, you can use the crawl delay directive in robots.txt. Please refer to the MSNBot support page for more information. Here are a few recommended settings:

Slow (wait 5 seconds between each request)

Crawl-delay: 5

Really Slow (wait 10 seconds between each request)

Crawl-delay:  10

Note that setting the crawl delay reduces the load on your servers, but it also increases the amount of time it will take MSNBot to index your website (proportional to the length of the delay), and possibly make it more difficult for your customers to find your site on Live Search.

Another great way to reduce the impact of MSNBot on your website is to enable HTTP Conditional GET and HTTP Compression as outlined in our prior blog post.

–Nathan Buggia, Live Search Webmaster Center

Join the conversation

26 comments
  1. Anonymous

    "you can use the crawl delay directive in robots.txt. "

    This is a useful fearture.

    We could use bandwidth efficiently.

    I wish if other search engines would support the same fearture.

  2. Anonymous

    Hi,

    Microsoft Live Search had not crawled my site for 3 months. Meanwhile Google and Yahoo had and I get visitors from Microsoft employees who use Google at work. Anyways, I removed the GZIP compression I was using and Microsoft Live now manages to get through somehow.

    Every browser and every other bot could deal with the compression. Seems like a bug in Live Search. Contact me at smallenucd@yahoo.com if you need me to help debug the situation. Thanks

    Sam

  3. rickdej

    Yahoo also supports this feature, but you’ll need to use the Google Webmaster Tools if you would like to ask Google to slow down their crawling.

  4. Anonymous

    I am glad to see that you’ve increased crawl rates, but I’d like to point out the lag between MSN and other search engines.

    * Google bot is on my site every few minutes, pretty much 24/7.

    * Yahoo is on my site a few times an hour.

    * MSN Bot- maybe once or twice a month.

    This is with a robots.txt discoverable valid sitemap containing 20,000+ pages (within protocol).

    I submitted my site to your 3 engines at the same time, and for the first time, last November 07.  Not surprisingly, Google is the only engine to have indexed most of my site, and where >90% of my visitors come from.

    I am using asp.net on IIS6 gzip enabled, conditonal get off, no code errors ever reported when MSN does crawl.

    I hope MSN doens’t always stay this slow.

  5. Anonymous

    Really happy to see the improvement, hope MSN Bot can come to my site more often. Most of my site are not so well indexed by MSN, hopefully after the implementation my site will get more attention from MSN.

  6. Anonymous

    My website disappeared from your Search result for 2 years causing me lot of money loss. If you can’t run a search engine please close it.

    Please give the address of the mathematician developed the algorithm for you. I want to sent 5 cents as my reward for inventing most crappy algorithm in the world.

    Regards.

    365greetings.com

  7. Anonymous

    Crawling slowliness is actually one of actual search engines failures. Any web-editor cant wait 5or more days to see one result

  8. Anonymous

    Currently only my main page is crawl, let see whether this improvement will lead to other of my web page crawled.

  9. Anonymous

    Although my home pages is index, but it only index 1/100 of my site.

  10. Anonymous

    Sam here, I started my blog three months ago and only the main page has been indexed while Google and Yahoo have indexed every article within a few days. Google even within one hour after publishing it.

  11. Anonymous

    Oh, I can control craw time of MSNBot! Great support!

  12. Anonymous

    My site is index in MSN search but is there a way to find the rank of your site?

  13. Anonymous

    great news, i should pay more attions about msn bot now.

  14. Anonymous

    This blog has been created to share useful information. Thanks and greetings! athough msn only index some of my site.

  15. Anonymous

    I have blocked your search engine’s access to my website, and since that day last week, addresses in the 65.55.0.0/16 range have registered  2223 packets and 107K of freaking data. If my bandwidth was metered, I would sue for recovery.

    You act like Chinese hackers, not a reputable search engine. If you simply obeyed the rules everyone else seems to respect for the robots.txt and UA strings that identify you as the searcher instead of an anonymous user, I might be willing to allow it, but every time you fools "improve" something, it takes six or seven years for you fix what wasn’t broken.

    Leave me the hell alone, please!

  16. Anonymous

    Please , Please, Please tell me how to restrict your bot from sucking the guts out of my bandwidth. I’ve tried a crawl delay of ’000′s of seconds but it seems to completely disregard that and return every few seconds. If this carries on I’ll have to deny your bot access to our site., and cancel our ads with you.

  17. Anonymous

    SAME PROBLEM AS ABOVE!!! I agree.

    Hello MSN? Anyone there? What is going on??

    “““““““““““““““““““

    Please , Please, Please tell me how to restrict your bot from sucking the guts out of my bandwidth. I’ve tried a crawl delay of ’000′s of seconds but it seems to completely disregard that and return every few seconds. If this carries on I’ll have to deny your bot access to our site., and cancel our ads with you.

  18. Anonymous

    I’m glad about the availability of these crawl settings, which I didn’t know about before. However, I prefer to leave things the way they are, to make sure that MSNBot crawls my site the way it wants.

    My robots.txt is in order and no spider is blocked, though I block the spiders from crawling the Admin panel. One issue I notice with MSN/Live Search is that it’s not indexing enough of my web pages like two other search engines do.

  19. Quality Directory

    I noticed the crawl improvement, but I don't have any need of implementing crawl-delay directive.

  20. get backlinks

    well Bing is faster than msn live :D

    so cheers

  21. Anonymous

    really useful info,that is good to know.

  22. raheelaquil

    how to define msnbot not to crawl or delay in robot.txt file?

  23. hamshahree

    hello

    i want to how i can change the crawling speed?

Comments are closed.