Introducing Bing Search Quality Insights

Dr. Harry Shum, Corporate Vice President, Bing R&D

I’ve been working in search at Microsoft for the last seven years and there has never been a more fascinating or challenging time to be in the search space. Today, search spans the entire spectrum of computer science from distributed systems and machine learning to natural language understanding and user experience design. The graph theory we learned in graduate school 20 years ago with a few dozen nodes and edges needs to be extended to cope with the web graph of billions of documents. There’s more web data created in a single day than in the entirety of 1999. Moreover, the types of data crawled by search engines are evolving in astonishing ways. What started as indexing simple web documents has blossomed into a dizzying array of data types – from rich multimedia content to real-time streams to social conversations, just to name a few.

In June 2009, we launched Bing ( with the aspiration of building a search engine that better understands what people have in their mind while searching so that we can help them get their tasks accomplished faster and more easily. The core to a great search engine has been and will always remain the same: delivering comprehensive, relevant and unbiased results that people can trust. We use thousands of signals from queries to documents and user feedback to determine the best search results and in turn make hundreds of improvements to our features every year, from small tweaks to core algorithm updates. As the web grows, we continue to take advantage of the collective wisdom from social, real-time, geospatial and contextual signals to further improve the search quality of Bing.

Over the last two and half years we have made tremendous progress in Bing search quality, measured scientifically by offline human judges and online user engagement. We have employed human judges to rate and compare search results from different algorithms. We also flight new approaches to a selected number of users to get online user feedback before rolling out new improvements. Search quality goes beyond scientific numbers such as DCG, nDCG, side-by-side, pSkip, however. The users often perceive the search quality by how fast the search results show up in a search engine, or how helpful it is to formulate the query e.g. auto-suggestion or type-ahead, or how easy it is to inform the clicks e.g. better snippets for the web documents. There is always so much more that users expect from search quality than what we understand and present.

With that in mind, today we are launching a new blog series we’re calling “Bing Search Quality Insights” aimed at giving you deeper insight into the algorithms, trends and people behind Bing. This blog is the first in a series that will take you behind the search box for an up close view into the core of the Bing search engine. Quality improvements in Bing are often subtle but often those little changes are the result of years of research. In the coming weeks and months, you will hear from members of my team on a range of topics, from the complexities of social search and disambiguating spelling errors to whole page relevance and making search more personal. We will also highlight the ideas and projects we have collaborated with colleagues from Microsoft Research and academia to advance the state of the art for our industry. We hope this will not only be useful information for our blog readers, but that they will spark conversations that help us all move the search industry forward.

Today, my colleague Jan Pedersen, Chief Scientist for Core Search at Bing kicks off the discussion with an overview of how we’re tackling the topic of whole page relevance at Bing. Jan delves into how we go beyond the traditional concept of page rank to deliver rich “answers” like video, images and maps that are relevant and help you get more done.

We would love to hear from you about what you would like to see covered so please join the conversation.

On behalf of the Bing team,

Harry Shum

Join the conversation

  1. nimageofmine

    This is a nice initiative, but am curious to understand your core purpose of this public blog. Is this aiming at educating the market by deeper insight into algorithms ? Are you expecting this blog to bring in more technical people into discussion to recieve constructive feedback/ideas thus improving the quality of Bing ?

    "The users often perceive the search quality by how fast the search results show up in a search engine" : I agree to this only to a certain extent. If the result pops up in 500 ms instead of 300 ms, I think it would be ok for a user. In my opinion, a user still looks for relevant answers (includes title, snippet, etc).

  2. rob-dean

    Will there be an RSS feed made available for this series?

  3. omrishabbat

    This is a great tradition you are starting here! It will surely help webmasters understand better how to optimize their site the best as possible.

  4. Andres S.

    Mr. Shum, Others,

    It is great to see an entry that exposes and discussed such an important topic as the search results quality.

    Such great enhancements have been made, like the social features, etc.  However, there is one subject of continuous grief from part of a very vocal group of users.  No official feedback has ever been provided:  Search results by date.  

    In fact, there is a very long -and resilient- discussion, unfortunately and mistakenly marked as "answered" by a sloppy moderator, whose first entry dates back to December 2009 and the latest entry, today.  

    This entry reflects the thoughts of people who care enough to discuss about the subject.  I can't help but wonder how many more are there that care about the subject but do not want to go trough the hassle of writing about it.

    Personally, I briefly remember seeing such a feature about a year ago -might be just an hallucination based on my wishes.-  Is the lack of date-filtered results a result of design decisions? Patents? Implementation issues?

    I hope this message helps (re)ignite the discussion. Thanks.

  5. NuNrg

    Excellent conversation and I find this blog very informative.  I will be delighted to understand more about the internal workings of Bing's search engine.

    To be honest, I am trying to understand my privacy issues more.  I was a devoted fan of Google even knowing that my information was being used anonymously.  The fact that it was being shared across so many different applications bothered me.  I dropped every single Google App including Gmail.  I am now taking a closer look at Bing to see if it has a tolerable level of use of my searches.

    I want to thank Andre for making me aware of this by linking to the many year old discussion on the need for Date filtering with Bing as the Search engine.  His comments are very well thought out, relevant, polite and concise.  

    I look forward to following this blog.

  6. budget3dmovies

    Great post. Google gives some tips on how there algorithms work and it is great to see Bing assisting now also. I have struggled to get high rankings on Bing and really need any tips on how to achieve this.

    Budget 3D Movies

  7. David Parak


    Very Nice

  8. Dr Geek


    We need to be able to readily filter search results by date. (last hour, day, week, month, year).

    When researching something that has happened recently I don't want to see search results that are 5 years old.

    We have been asking for "filter by date" (or narrow by date as you might call it) ever since Bing arrived but no-one is listening.

    Until that functionality is available I will have to keep using Google search even though I don't want to.

  9. tempvisit

    I agree with Dr Geek. I search a lot regarding software development, and need to filter based on dates.

    Why it is not implemented is beyond me. Is it too complicated? I am not using Bing for this very reason.

    And as NuNrg writes, Google's new privacy "agreement" has turned me off.

    I am now seriously considering using Tor on a permanent basis…

  10. mosley

    Thank Harry, i will be reading all of these.

  11. kev1h

    still looking for search by date, can't switch from Google until this basic feature is added!

  12. jjb2012

    Date filter, date filter, date filter…. and I switch to Bing, recommend Bing, "Bing"! Why not? MS is _so_ slow and sometimes they never add something that would make me want to switch.

Comments are closed.