One of the key elements of having a great index is having fresh content.  If someone searches for Vista on the day that we announced the name for the next version of Windows then they should be able to see relevant results.  This means that we need to have the content in our index within minutes of these types of events happening. In order to do this we built a simple crawler to get News and Blogs that break this type of news.

As you may have seen from Jeremy Zawodny’s blog, Greg Hughes blog and others we are crawling RSS and other content that we want to be sure is always fresh in our index.  At this point we are just experimenting and suffice to say that in our short experiment we have learned a lot.  We have been a bit more zealous in our crawling then we would have liked and we are going to fix that.  Thanks for the feedback!  Stay tuned for more as we figure out what to do with all of this data…

Eytan Seidman, Program Manager, MSN Search