Bing and cloaking detection

Since the inception of the Bing Search team a few years ago, we’ve been maniacally focused on one thing: relevance. We use the term relevance to mean that Bing finds the answer to your question better than any other source – whether your question is best answered by a website, or a real-time traffic map. And with our recent Fall update our relevancy has improved dramatically – to the point where we think we’ve got the best product in some areas, and a highly competitive product in others. (Try it out, I dare youimages, maps, mobile, web)

One of the biggest challenges with relevancy is how to distinguish legitimate information from various forms of search spam. This is one area that we’ve made especially good progress in over the last 8 months through a suite of tools that helps us detect, evaluate and manage spam. One of these tools is an extension to BingBot, giving us an additional way to detect cloaking. (It should be noted that not all cloaking is spam related and we do our best to take this into account, however, we still don’t recommend cloaking in any situation).

The goal of the tool was simple, however there have been some well-documented short comings in our implementation that have impacted the reporting metrics of some websites. We have been listening to the feedback over the past couple months and continuing to optimize the tool to eliminate these issues:

  • AdSense/Overture reporting – Initially there was a bug in our crawler that caused it to download all content on your page, including ad blocks. We have since fixed this issue by blocking requests to Google and Overture to preserve the integrity of your reporting.
  • Distort site statistics with unfilterable bot traffic – Webmasters have also reported a high level of traffic coming from this bot, in some cases high enough to impact their logs in a statistically significant way. We have been continuing to optimize the crawler and most webmasters should notice the referrer traffic dropping to almost nothing over the next month.
  • Pollute HTTP logs with inappropriate terms – Another unfortunate issue is that we were using a common list of keywords for our testing that was not site specific. We have tuned this list and you should no longer see any keywords used that are not related to the content of your site.
  • Microsoft isn’t responding to questions –  Webmasters who encountered these problems and reported them to Microsoft have not been able to get a satisfactory or timely response. We have created a forum specifically to answer your questions and comments. For sensitive issues, please use our feedback form to contact us privately.

Hopefully webmasters have also noticed these issues disappearing. If you are still experiencing any issues, please contact us before you block BingBot, to see if we can address the issue.

We’re inspired by the relevancy improvements we’ve made this Fall, and have much more in the works for Spring. Please keep us in mind as you do your searching, and let us know how you think we can make search better.