Live Search is listening

In a blog post last month, we identified the preferred method for providing feedback on any Live Search crawler issues you may have experienced. We would like to thank everyone for taking the time to provide us with this feedback and let you know what we have done on our end. But first, a little background.

Our crawler, MSNBot, performs many different functions. We have blogged about our cloaking detection in the past, so I’m not giving away any secrets by saying that we are actively looking to identify and weed out spammers who are using cloaking in malicious ways. We’ve also introduced a new feed crawling function that will help provide fresh results. In addition to this, we are also introducing a new 2.0 version of our crawler, MSNBot, which has various other new functions as well as upgrades on some of the old functions.

So, what I’m getting at is, we are busy adding upgraded and updated technologies to provide better search results. That’s good news for you!

Unfortunately, things can and do go wrong from time to time. The initial complaints, that we were over-crawling some servers with our cloaking detector, was compounded by and also confused with the new release of our feed crawler that was also overzealous in its attempt to crawl and provide up-to-the-minute results.  However, we have taken all of the feedback you have provided and made some improvements.

We have modified the cloaking detector. Using the valuable feedback we received regarding the feed crawling issues, we proactively released a patch late last week that should significantly reduce the number of requests to a more acceptable rate.

What you can do

Help us discover the content changes on your site. You can do this via sitemaps and various meta properties per link or via RSS link to notify us about very important content. To prevent us from having to monitor lots of feeds often, we recommend aggregating content change onto a few feeds; adding the name, “Aggregate” somewhere in the feed name. We also suggest referring to them in robots.txt and your sitemap—both of which will help us detect them and their use.

Despite releasing a patch for our feed crawler, not all sites are the same, so it is a challenge to gauge a feed crawl-rate that is considered reasonable for all sites. If you believe we are still crawling more than necessary, an alternative option would be setting a crawl-delay. We would urge caution while setting the crawl-delay times as they can severely hamper our ability to crawl your site. As an example:

A crawl-delay: 5 means we wait 5 seconds per page request. That means if you have a site with 100 pages, it will take us 8.3 minutes to crawl your entire site. A site with 1,000 pages will take 83 minutes. As you can see, this can severely hamper our ability to index fresh content from your site.

If you are using crawl-delays and we don’t seem to be honoring those directives, please do not block us. Instead, let us know about these issues and we will work together to find out why we aren’t honoring the crawl-delay directive.

For all of these issues, please send us feedback by posting a message in our Indexing and Ranking Discussion forum. Thank you once again. We look forward to working with you in the future!

–Brett Yount, Live Search Webmaster Center

Join the conversation

17 comments
  1. Quality Directory

    I don't have any crawling issue. And it's pleasing to hear that you have upgraded and updated your search engine to bring better search results like Google and Yahoo.

  2. Anonymous

    I like bots to crawl my shop and I make a sitemap.txt so that it would be easier for them to find my pages. I don't like too much though when Bing gives my human visitors my sitemap.txt – how does one turn off that ?

  3. fortisx

    I haven't seen any method to allow Bing to find one's site. My site is always close to the top on Google for the main search phrase in my industry. But my site doesn't appear anywhere on Bing.

     I have not found any way to tell the Bing crawler about the existence of my site. This is pretty basic stuff, e.g., providing one's URL, but MS has not thought of it yet.

  4. lwarren

    We have a nice bing listing for our hotel, Hadsten House Inn & Spa, in Solvang, CA.  

    However a wrong web site link (www.vagabond.com) is listed as the web link for our hotel in error.  How do we change the incorrect web link to the proper and correct url of http://www.hadstenhouse.com

  5. bill_the_catz

    I own a news network yet none of our content shows up in Live/Bing news.  How can I get our web content added to the news area?

    Also, how can I get crawled more often? We update daily yet our crawls are every 2-3 days.

  6. get backlinks

    today i have uploaded my sitemap and submitted to bing, lets see when it crawl my website

  7. Anonymous

    Hope to be better. Better means more features.

  8. Anonymous

    I'm counting on you.

  9. Anonymous

    Do you have crawling rate option? Like what google did on their webmaster tool

  10. Anonymous

    Good to see Bing honors the crawl delay option, was having some serious problems with a set of sites being 'over' indexed by Google (kept crashing a dedicated server) and there was no simple on site way (like crawl delay) to slow it down!

    I own over 100 domains and didn't want to setup 100 Google webmaster confirm a site thingy :) for something as simple as this.

    David

  11. Anonymous

    My site got crawled like 2 months after I submited it and now they've only indexed 1 page in 3 weeks since then!

  12. David Law

    @Undercrawled Submitting a site to any search engine is basically a waste of time, all major search engines rely on links which is also how they find your site for crawling.

    Basically if no sites link to you (a vote saying this content is good) why would a search engine want that page? So the most likely issue is a lack of backlinks.

    David Law

  13. Blackpool UK

    i agree with david law's post on this relating to the issue of backlinks.

  14. miles2go

    MS team working on making it a better search engine which can be seen easily.

  15. hotels

    bing is getting better with their search results

  16. Anonymous

    Want to be better. Better means more features

  17. reactivenonsense

    I own a news network yet none of our content shows up in Live/Bing news.  How can I get our web content added to the news area?

Comments are closed.