Looking for Input on Crawl Delay

In keeping with our themes of rapid change and responding to input and feedback from the webmaster community, we are working on future feature planning for the Bing webmaster tools and we would like to hear your thoughts in a few areas.  The first area is Crawl-Delay:


Today, we fully support crawl delay which is specified in robots.txt.  We are considering creating a tool which allows you to directly control the frequency, and perhaps even the timing, of msnbot/bingbot.  A few questions:

  1. Do you prefer for us to continue to support crawl-delay in robots.txt?

  2. Would you like to be able to control the crawl rate based on time of day?    
    1. What are the ranges of crawl rate you would like to see if you had control over the time of day? 
    2. Would you want to turn crawling on/off by time, or just specify different crawl rates based on the time of day?
  3. If you had this ability to directly control the crawl rate through a webmaster tool, how often would you adjust the settings?
  4. If we built this tool, would you like it to control our fetching directly, or would you rather have the tool output the proper robots.txt entries which you then insert into your robots.txt file?

 Feel free to reply to this post, or send email to bwmc@microsoft.com.


We thank you in advance for your feedback. 


Share on twitter Share on facebook Share on linkedin Share on linkedin