Thoughts on search quality

This morning, I will be on a panel at the Farsight Summit with some of the
industry’s thought leaders to talk about search quality as we look into the
future of search. Farsight is about looking 10 years into the future to explore
the big industry challenges and opportunities ahead of us as we work to deliver
the next generation of search.

We woke up to an interesting (and interestingly timed) article
by Danny Sullivan about some complaints Google has about how it says Bing ranks
results. I wanted to take a moment to make a couple of points in advance of this
panel so we can stay focused on the original intent of the Summit.

The Bing engineering team has been working hard over the past couple of years
to deliver the best search relevance and quality in the industry and for our
users. This is our top priority every day.

We use over 1,000 different signals and features in our ranking algorithm. A
small piece of that is clickstream data we get from some of our customers, who
opt-in to sharing anonymous data as they navigate the web in order to help us
improve the experience for all users.

To be clear, we learn from all of our customers. What we saw in today’s story
was a spy-novelesque stunt to generate extreme outliers in tail query ranking.
It was a creative tactic by a competitor, and we’ll take it as a back-handed
compliment. But it doesn’t accurately portray how we use opt-in customer data as
one of many inputs to help improve our user experience.

The history of the web and the improvement of a broad array of consumer and
business experiences is actually the story of collective intelligence, from
sharing HTML documents to hypertext links to click data and beyond. Many
companies across the Internet use this collective intelligence to make their
products better every day.

We all learn from our collective customers, and we all should.

From its inception, we have had what we believe is a distinct approach to
search, and the features and innovation in Bing – from our new user experience
and visual organization approach to our focus on inferring user intent and
helping customers complete complex tasks, Bing has added a new voice and new
experiences to search. We never set out to build another version of an existing
search engine. We believe search needs to do more for customers. This is the
guiding principle in how we approach our work each day and it is why we are here
today engaging in discussions at Farsight, an event we are proud to have helped
bring about.

This is why I am excited to mix it up this morning with some of the best
thinkers in the search business.

We all have lots of work to do to get search to be everything we want it to
be, and everything our customers deserve. Let’s get to it.

Harry Shum, PhD, Corporate Vice President, Bing

Join the conversation

57 comments
  1. parsonsproject93

    I have to say, this entire fiasco is destroying Bing's reputation, I'm having relatives e-mail me about this just to take jabs at Bing, because I told them that I really like Bing.  People now think that Bing is basically copying all of Google's results.

  2. randfish

    Harry – I'm glad to see Bing being transparent about data usage and honestly, I think this is the right call. When a market leader has a 7-year headstart, I fully expect that you should use every arsenal in the toolbox.

    This is, to some degree, within Google's rights to complain about. You're using their data in a smart way to improve relevancy and they built a nice honeytrap to expose it. That said, the timing of their release made them look really bad. To sit on info like this for 30 days, waiting until they knew you'd be in the room together and employing the news to deflect from their own spam/relevancy issues just isn't cool.

    Here's to hoping that we can be more professional and authentic about criticism, praise and search quality issues in the future on both sides.

  3. vkelman

    So, you basically admitted you're spying on Google Search to steal their quality results.

    It's not surprising, they proved it, so you cannot say you didn't do that.

     Congrats!

      http://www.google.com/…/vkelman

  4. glopal

    @randfish

    Why would someone put an arsenal in a toolbox? You really need to work on your metaphors. As far as shifting blame goes, you got that one down.

  5. asdasd

    In 1000 different signals you have you've picked the worst one. Direct copying of results is just below anything.

    @randfish: yes, MSN Search/Live/Bing had 7-year start as a market leader (actually, I thought it was AltaVista?) but now they aren't. Now we have only what is, essentially, a Google carbon copy, an imitator, an imposter. Sad.

  6. parsonsproject93

    @asdasd

    But it's not a google carbon copy, like the post said, using user habits to index is only one way that it indexes.  It doesn't use the Google Search results to say, "Ok, so this is on Google's front page, I should now put this on Bing's front page".  It simply indexes the site.

  7. webguy

    Well played Bing team :)

  8. randfish

    Those who think this is "direct copying of results" are mistaken. Google's spinning the PR that way, but that's not what happens. Bing uses signals like clickthrough on SERPs in their 1000 factor algorithm. When you remove any other signals (which is what Google's honeypot did), all you're left with is the 1 Google thought they catch MS using, and that's what happened.

    I know it's compelling to attach onto the simple "Bing stole from Google" story, but like everything else in the media, it's a vast oversimplification. I'm hard-pressed to believe that anyone who was tasked with beating Google at search wouldn't employ this tactic if they had access to it.

  9. privacy-alert

    "But it doesn’t accurately portray how we use opt-in customer data as one of many inputs to help improve our user experience."

    You mean how Bing uses IE/toolbar user's personal communication between the user and a 3rd party website? which you opt-in for them by default?  Yeah, your portraying of the story is more accurate alright.

    If every arsenal in your toolbox is fair game, then why don't you just redirect all "www.google.com" url to "www.bing.com" in IE while you are at it.

  10. Rizarsky

    So when are we going to start seeing "Powered by Google" on Bing results?

  11. johnnliu

    Many companies have internal database-based search for the products on their catalog.  So if a customer is using my own search and locates a product in my catalog, and happens to have the Bing toolbar installed, then Bing Toolbar will pass that information to Microsoft.

    Then next time someone asks Bing about that product, Bing will be able to direct them directly to my product page.

    So suddenly OMG Bing is copying my search algorithm?  And this practice should stop?

    Personally I really don't see what's wrong with this.  In fact, if the data helps Microsoft identify why I couldn't find something in Bing but I could with Google, so they can tweak their query and improve the results, I'm all for it.

    The whole "copying" is a really lousy marketing spin.

    @privacy-alert

    When is the last time IE automatically installed the toolbar or opt-in by default?  Have you actually done this recently or are you just prejudiced to begin with?

  12. Cornario

    So if I took years of R&D to make the algorithm that can find your result, and the other company can just use my result and display as their own?  I don't think so.  If Bing can't find something for you.  They should just improve their algorithm so they can find the result.  Get result from others is low.

  13. privacy-alert

    From the orignal article by Danny Sullivan, linked above.

    "When you install the toolbar, by default it is set to collect information to 'improve' your experience, as you can see:"

  14. experiencemusic

    A more accurate discription of this approach might be that Bing is using the click behavior of google users to improve Bing results – not google results. There is an important distinction in that statement. Bing is not copying Google or their results -rather It is learning from user behaviors across several sites, one of which might include google. Another example might be if lots of people are visiting a breaking news article, one could imagine Bing might consider ranking that article higher in their news section based on user click behavior/interest. Yet, that would mean that Bing is copying the ranking of news sites, who might also have that result at a top spot on their homepage.

  15. shederman

    I wonder how you'd feel if your child got caught cheating in a test at school and told you "but I only copied a couple of the answers". Microsoft's behavior is outright cheating, copying the other kids results, arguing about the degree of cheating just makes you look like a guilty schoolchild in front of the headmaster.

    blogs.palantir.co.za/…/bing-caught-cheating.aspx

  16. illegaloperation

    What is happening is that Microsoft is monitoring search box on every site. If 500 people go to "example.com" and enter "item X" into the search box and most click the second link which leak to "news.com", Microsoft would add that link to Bing. Suddenly searching "item X" on Bing would result in "news.com" as the first result.

    I don't see anything wrong with that, but Google need to complain because it can't take the heat from competition.

  17. illegaloperation

    What is happening is that Microsoft is monitoring search box on every site. If 500 people go to "example.com" and enter "item X" into the search box and most click the second link which leak to "news.com", Microsoft would add that link to Bing. Suddenly searching "item X" on Bing would result in "news.com" as the first result.

    I don't see anything wrong with that, but Google need to complain because it can't take the heat from competition.

  18. Ghrowe

    While I understand the explanation of the use of clickstream algorithms to enhance search results. I don't think it's a great indicator to solely base results on. If the clickstream data is the only information Bing has it may not be a reliable result, as has been demonstrated by Google with the gibberish strings.

  19. nagaharish.movva

    BING – "Because It's Not Google"

  20. iAndroid

    Then you have to credit the results which belongs to Google (and other search engines) but you did not. This is outright plagiarism. #FAIL

  21. experiencemusic

    @ shederman – in your analogy, it seems bing and user click streams would be a study group and not blatent cheating on a final exam. Remember, it was reported that only 7-9 out of the 100 synthetic searches google created actually saw a mirrored behavior on Bing. This is obviously not hardcoded into Bing, but "1 of a 1000" various ways used to improve bing results.

  22. OneLag

    "We use over 1,000 different signals" – in this case, your only signal was the google search result. I'd tottaly understand if the google search results were only ONE of the thousand signals, but that's not the case there. That's the great problem in my opinion. Those search results were just copied.

  23. OneLag

    "We use over 1,000 different signals" – in this case, your only signal was the google search result. I'd tottaly understand if the google search results were only ONE of the thousand signals, but that's not the case there. That's the great problem in my opinion. Those search results were just copied.

  24. Tanishq Goyal

    I have say that Google is really getting afriad of Bing now.

    And the way Google's post is put up, any layman just belives that we directly display Google's top results on Bing page.

    They can't or may be don;t want to note that the similarity was found only in 10 cases out of 100 and thesecases were special honeypots.If you look at common queries, you will see that Bing has far better results than Google..

  25. SemSemSam

    Shame on you Microsoft, Shame on you.

  26. SemSemSam

    Shame on you Microsoft, Shame on you.

  27. SemSemSam

    @Tanishq Goyal you are an idiot and you know it.

  28. Jusfitu

    I like ‘Bing’.

  29. Gopinathp

    LOL. So instead of searching and indexing the web, you take shortcuts like this.

    We'd Like For This Practice to Stop or mention "Bing: Powered By Google™" in your home page.

  30. SemSemSam

    "We use over 1,000 different signals and features in our ranking algorithm" and one of them is stealing search results from Google, well done Bing Team

  31. SemSemSam

    "Bing: Powered By Google™"

  32. SemSemSam

    Wait a second, if what you say is true that you stole Google search results to "enhance your search results" then why do you serve results directly to your users? Shouldn't a data collection practice be used only internally to study quality and then amend your results accordingly? I really wonder what new lies the Bing (Powered by Google) Team would come up as an answer to this question.

  33. Searcher99

    For all those people slagging Bing for "stealing" Google's long-tail search results, what are your views on Google copying Bing's innovative visual search feature, as well as copying BIng's webpage preview that pops up when you over the search result link? Pot calling the kettle black a little bit.

  34. SemSemSam

    @Searcher99 How about Microsoft stealing Google search engine by coming up with Bing in the first place, the way they laid it out completely copied Google, if it wasn't for Google, you would've never seen MSN search, Live, Bing or any other lame name they come up with.

  35. suskunsair

    Those who think this is "direct copying of results" are mistaken. Google's spinning the PR that way, but that's not what happens. Bing uses signals like clickthrough on SERPs in their 1000 factor algorithm. When you remove any other signals (which is what Google's honeypot did), all you're left with is the 1 Google thought they catch MS using, and that's what happened.

    I know it's compelling to attach onto the simple "Bing stole from Google" story, but like everything else in the media, it's a vast oversimplification. I'm hard-pressed to believe that anyone who was tasked with beating Google at search wouldn't employ this tactic if they had access to it.

    <a href="http://www.suskunsair.net"title="sesli chat"target="_blank">sesli chat</a>

    <a href="http://www.suskunsair.net"title="seslichat"target="_blank">seslichat</a&gt;

    <a href="http://www.suskunsair.net"title="seslisohbet"target="_blank">seslisohbet</a&gt;

    <a href="http://www.suskunsair.net"title="sesli sohbet"target="_blank">sesli sohbet</a>

    <a href="http://www.hasretgulu.com"title="sesli chat"target="_blank">sesli chat</a>

    <a href="http://www.hasretgulu.com"title="seslichat"target="_blank">seslichat</a&gt;

    <a href="http://www.hasretgulu.com"title="seslisohbet"target="_blank">seslisohbet</a&gt;

    <a href="http://www.hasretgulu.com"title="sesli sohbet"target="_blank">sesli sohbet</a>

  36. SemSemSam

    @SuSKuNSaiR Stop spamming this place with your crappy site.

  37. Searcher99

    @SemSemSam – It might be time for you to calm down a bit and acknowledge than Google's holier than thou philosophy has been dead for years. As for your insinuation that any search engine is just a poor version of Google, or that launching a rival search engine is in fact copying Google, well that's laughable. Obviously the fact that Google completely mimicked Bing's revised search results layout over the past year is totally different to copying a competitor. Or the fact that if someone uses the bing search engine when using Chrome, Google will hijack every single piece of data from that experience to improve their own algorithm? Portraying one search engine as the devil and another as the innocent pinnacle is so one-eyed it's untrue.

  38. SemSemSam

    "Google will hijack every single piece of data from that experience to improve their own algorithm?" A poof link please? a study, a honeybot? talk is cheap, and that's what you are doing now, you seem to share this same tactic with Bing (Powered by Google) Team, coming up with crap that does not exist. Not only did you commit this shameful act stealing people's hard work, you also defend your self as nothing has happened.

  39. Searcher99

    @SemSemSam – Yet again you're continuing to ignore Google's own misgivings when it comes to imitating other search engine features. Your deliberate ignorance is depressing. How about calling Google "Google (Powered by every great start-up company that we've been lucky enough to buy and then ruin before they become our competitors")? It's naive to think that Google doesn't use data from other search engines or open areas (streetview wifi data anyone?), and it's well known how Google rips open the content of your Gmail inbox before you've received any mail to learn how to demographically target you better with their advertising.

    If you want a better balanced article on what this "copying" saga means for both Bing and Google, give this a read:

    blog.searchenginewatch.com/110201-211436

    It's certainly better balanced than Danny William's article on Searchengineland

  40. SemSemSam

    @Searcher99 I know my computer skills are very very poor, I just can't seem to find anything related to Chrome hijacking user data when searching on Bing on that lame article you sent, could you please help me out here?

    You are so desperate to talk about Google collecting wifi data, try to steer away from Bing (Powered by Google) Team Shameful thing to something completely irrelevant, at least Google admits their mistakes to the public, unlike Microsoft, steal competitors data and remain silent about it.

    P.S you must be a Microsoft employee to have this blind trust in such an unethical company, wait a sec, no, even Microsoft employees don't like Microsoft.

  41. Talal Masood

    Since the launch of BING, I am using Bing, and had always been comparing results from Google. I found bing's results much better and to the point. Google has a lot of junk results which are not what exactly you are looking for. I would suggest everyone to try searching for something on Google and Bing both, you will most probably like Bing's result :)

  42. metamezzo

    "We use over 1,000 different signals and features in our ranking algorithm. A small piece of that is clickstream data we get from some of our customers, who opt-in to sharing anonymous data as they navigate the web in order to help us improve the experience for all users."

    Did Google generate this clickstream data also therefore triggering this "small piece" of Bing's ranking algorithm? If so, then truly "spy-novelesque". If not, then there's Google's point, however "interestingly timed".

  43. z-u-r-l-i

    So c'mon !!! One of the worst thing seen from Microsoft so far …  how can you think about using Google data to improve your serps ? And then blame on google being full of spam ? So you are a copy-spammer ?

    If you don't have enough brain to think of good algorithm just tell it. We are not good enough !

  44. Quaz and Wally

    I feel much better knowing that Bing copies search results from everybody and not just Google!!!  That is certainly much better than just stealing Google's search results!

  45. Middleman

    Let's all be honest –  Aren't we tired of Google monopolistic attitude in Search – EU investigating Google on unfavaroble treatment to "known" brands. Bing as a search/decision enigine and like any other has signals to improve the relevance and quality of search results which apparently Googles results are used!  Holy Moses ! Let's encourage other competitors to improve our search results rather than criticising them. Google as a company are worried about losing their market share to BING. What would happen if Bing becomes the #1 search engine? Google will spot a loopwhole to get them done!

    My point: Search engineS should actually work together to improve the quality of search results – PERIOD!!

  46. nimageofmine

    such a immature statement from a matured company google. Sad to see that Google does not like competition and good to see that Bing is getting into its nerves.

  47. Patthebrat

    If as Bing mentions they thrive to get the best search results for their customers shouldn't their algorithm detect that a "user feedback data" is plain wrong as it was the case for the honeypots google setted ?

    Otherwise, shouldn't they be afraid of results being faked by clickbots ?

  48. HRM

    Mr.Wheeler, Yes, I copied John's answers, but that is only 1 of the 10 people I copied from, and remember, he copied ALL his answers from a textbook!! (by studying day & night – but I dont have time to do all that) I am the underdog, so I use all the tools available at my disposal to come up with the answers – as we all should.

    That is why I am excited to mix it up this morning with some of the best thinker in the exams.

  49. fsck

    Why am I not surprised. I bet Microsoft never took ideas from Apple too, right? Suure… Just like they always cared about security – even while ignoring backdoors (backorifice 2k anyone ?) that were shown to give remote access [hence back door!]. C'mon Microsoft, what happened to innovation that doesn't come down to petty theft of ideas ?

    Windows 7 is about the best thing from MS since Win 3.11 and that's saying something!

    And before anyone says they're not Microsoft do realize this :

    Administrative Contact:

           Domain Administrator

           Microsoft Corporation

           One Microsoft Way

    (from whois of bing.com).

    And yes, I'm being a filthy troll who doesn't use Microsoft products (or apple, admittedly) – thankfully!

  50. arthurduarte

    Instead of just repeating what's being said in some websites that worry more about page views than with accuracy, why don't people just don't search from themselves and see what happens? Playing the blame game is really easier, I guess…

    Has anybody here ever had any statistics class? Than you should be ashamed of yourselves for believing in statistics based upon singularities, which is basically what Google is trying to feed us. And ain't the timing perfect? They blame Microsoft for something that can be easily disproved (but that makes a mess, either way) and diverges public attention from the growing number of spams in Google's search.

    Wake up, people. This is not about heroes and villains. It's about a huge corporation trying to attack its main competitor, with some 'Scooby-Doo'-esque type of "investigation".

  51. Ectogamut

    Google voluntarity opted in to provide the clickstream data for the "honeypot" queries to Bing. If they had concerns about Bing using that data, then they should not have opted-in in the first place. Doesnt Google know how to decline a EULA after the "thousands of person-years" that it has spent? It is not that complex – send me a Benjamin and I will show them how it is done. What did Google think that Bing would use the clickstream data for – definitely not to print it out and use as wrapping paper. Of course Bing would use the clickstream data, whether it comes from Google or whether it comes from John Doe. Google cant be that naive – or could it?

  52. Horse Mouth

    I do not remeber that Microsoft has ever exhibited the syndrome of te mania of greatness by deleting, closing and ignoring inconvenient questions and feedbacks like:  

    "When google search gives incorrect results – how can it be reported? [closed]"

    http://goo.gl/YJdtJ

  53. Guinevere

    The approach of skimming google's successful search results and directly injecting them into Bing's search index can only work as Google is better than Bing. Why don't google do this too? Because currently, they have the search methodology most people are happiest. They don't need to copy the "hard to index" stuff from someone else.

  54. Jake L

    "Imitation is the sincerest form of flattery."

    Proverb: Copying someone is flattering because it shows you want to be like that person.

  55. x_Jiro_x

    "We use over 1,000 different signals and features in our ranking algorithm. A small piece of that is clickstream data we get from some of our customers, who opt-in to sharing anonymous data as they navigate the web in order to help us improve the experience for all users."

    From what i understand…

    Bing does not directly copy results from others but only collects best search results it receives from users.

    … this is still cheating.

  56. hotels

    I like Bing!

Comments are closed.