Most search engines still provide a fixed number of web links on the search results page, typically 10 blue links in addition to instant answers. But are 10 blue links the right number of links for all queries and scenarios? Intuitively it doesn’t seem so. For example, showing fewer search results in navigational queries may enable users get to their desired URL faster. Also, users may benefit from seeing more blue links when they return to a search results page after clicking on the browser’s back button. In this blog my colleague Ronny Kohavi explores the benefits of having a dynamic number of search results and provides valuable insights that led Bing to challenge the status quo.
– Dr. Harry Shum, Corporate Vice President, Bing R&D
At Bing, our central mission is to help users spend less time searching and more time getting things done. To that end, we run over 100 concurrent controlled experiments at a given point in time, looking for novel ways to improve the user experience. While most ideas fail, and the statistics are humbling (e.g., see this talk), this blog showcases an interesting example of ongoing changes we make to Bing.
Since the early days of modern search, the search engine results page, or SERP, has been largely defined as ten blue links. Outside vertical searches (e.g., image search) and instant answers (e.g., weather, and how tall is tom cruise), little has changed on the size of the algorithmic part of the results page. We recently challenged the conventional wisdom around this notion, and were able to show using controlled experiments that ten blue links are not the most useful way to present results; rather, we adapt the search results page size dynamically.
In general, the further a link is from the top of a results page, the lower its click-through rate. This is due to two factors: the search engine optimizes more relevant items higher, and the users’ scan pattern (in the US) is from the top going down.
On average, over 50% of users click on the first result on the page. From there we see a significant drop, with less than 1% of people clicking on the 8th link on the page. The figure below shows the click-through rate dropping from position 3 on, where position 3 could contain an Instant Answer, or the 2ndweb result that was pushed down because there was an Instant Answer above it, etc.
With this insight in mind, we looked for interesting cases where the click-through rate is much higher on lower results. One interesting case was after a user hits the back button. When users click on a result, then hit the browser back button, they typically look lower on the page. Statistics showed that the click-through rate on lower positions are a factor of five to eight times higher after a back button. This observation led to a change to Bing in the US in late May 2012 so that the SERP initially showed eight algorithmic results, and the page was extended to 12 after a back button. The controlled experiments showed that key metrics improved: users were executing fewer queries per session, pages rendered faster on average, and pagination to page 2, 3, etc. reduced by almost 2%.
Continuing this line of thinking, we asked the opposite question: are there cases where the click-through rates decline so quickly, that it’s not even worth showing even eight results. Does one really need to see eight results for the common query “ebay”?
These queries have “deep links” and the click-through on the first result, a large block, is over 75%. The decline in click-through rate is so dramatic in these cases, that the 3rd algorithmic results has a click-through rate lower than 1%, i.e., lower than the 8thalgorithmic result on a regular results page.
We experimented with truncating such result pages more aggressively, showing only the first four elements (either algorithmic results or instant answers) and news when relevant. Conversely, we extended the SERP after a back button to 14 algorithmic results. The results were great and our key metrics improved: users were more successful in sessions, they reached the web link they wanted sooner, queries per session declined, pages rendered faster on average, and pagination was reduced by a further 5%. The feature shipped to all users on April 22nd, 2013.
Where do we go from here? These are our first steps in dynamically sizing the page. The steps are small, using the MVP (minimum viable product) approach. We continue to challenge the status quo and we are now looking at ways to determine whether we should extend the number of search results when we have evidence that users are in exploratory modes (e.g., searching for insurance). If you’re getting different number of results, you’re probably in one of our ongoing experiments.
– Dr. Ronny Kohavi, Partner Architect, Bing R&D