There has been a significant amount of media discussion about search and malware in the past week following the release of AV-TEST’s report on the topic.  The study claims that Bing exposed searchers to more malware than Google, meaning when a searcher enters a query, they are more likely to go to a link that could potentially hurt their system.  Many have wondered why we have been so quiet over the past week on the topic given the potentially damaging claims being made.  In truth, this is a complicated topic and we were doing the research necessary to make sure we understood both the study and the true scale of any reported issue with our search results. Unfortunately AV-TEST wasn’t able to provide their methodology or many other details until Wednesday, but one thing is clear given the information we have: AV-TEST’s study doesn’t represent the true experience or risk to customers.  In other words, the conclusions many have drawn from the study are wrong.

Why?  AV-TEST didn’t actually do any searching on bing.com.  Rather they used a Bing API to execute a number of queries and downloaded the result to their system for further analysis.  By using the API instead of the user interface, AV-TEST bypassed our warning system designed to keep customers from being harmed by malware. Bing actually doesprevent customers from clicking on malware infected sites by disabling the link on the results page and showing the below message to stop people from going to the site. 

You may ask why we show these links at all if we think they are infected with malware?  We don’t explicitly remove malicious sites from the index because most are legitimate sites that normally don’t host malware but have been hacked.  Our research shows that if sites like this remain infected for a long period of time, their ranking will naturally fall because customers won’t click on them.

We warn our customers rather than suppressing the result for both completeness and educational reasons: first, if a user searches for “vacation hotline” and doesn’t get the site they’re looking for they perceive Bing to be an incomplete index of the web which impacts their confidence of the engine; second, if they can’t find the result on Bing (because we removed it completely from the results) and then they go to Google, search for it, find it, then click on it (because Google may not have detected it as malware) their machine could be put at risk.  With Bing’s warning, they know “oh, okay, I should probably try again another time”. 

Indeed as of this writing, if you search Google for “vacation hotline”, you will see the potentially infected result returned by Google and if you click on it, an attempt could be made to infect your machine. To help mitigate this scenario, we warn customers about potentially malicious links and our data shows that these warnings block 94% of clicks to malicious sites.

Does this mean Google is bad and Bing is good?  No, it means this is a highly complex problem that all engines are constantly working to solve.  No engine will be perfect 100% of the time but we all work every day on detecting the latest threats from the bad guys and updating our engines to keep customers safe. 

One of the tricky things about malware detection is that the malware distributors are playing a constant game of cat-and-mouse with search engines and webmasters.  This means they employ various techniques to hide their infections from web crawlers, searchers and the site owners based on all sorts of factors, such as location, behavior, operating system, and which search engine the user browsed from.  The end result is that malware may not always be presented on every visit or the webmaster may have discovered the infection and cleaned it up. At Bing, we take signals from our detectors that might indicate recent malicious behavior and use our warning system to alert customers that the site might present a real risk to them. At the same time, websites that register with Bing’s Webmaster Tools are alerted that we may have found malware on their site.

We show results with warnings for about 0.04% of all searches, meaning about 1 in 2,500 search result pages will have a result with a malware warning on it.  Of those, only a small proportion of malicious links ever get clicked and the warning therefore triggered, so a user will see the warning only 1 in every 10,000 searches. In any case, the overall scale of the problem is very small.

While independent studies on how malware attempts to use search engines to spread are helpful for vendors and customers to understand how to stay safe online, AV-TEST’s methodology doesn’t tell the whole story; in the modern world of information sharing, it’s not enough to simply block everything outright, rather keeping customers informed and able to make the right decisions is key. We are working with them to help deliver more accurate and real-world results for such future tests. In this particular case, we here at Bing are very confident that our methods for malicious link detection and warning make our engine one of the safest on the net.

Happy and safe searching!

– David Felstead, Senior Development Lead, Bing