A few months ago we provided a behind the scenes look into how Bing is improving image search quality. In this post we wanted to explore some additional techniques we are employing to deliver high quality image results. My colleague Eason Wang will give you a closer look at how we are incorporating aesthetics to deliver more beautiful image results in Bing.
– Dr. Jan Pedersen, Chief Scientist, Bing and Information Platform R&D
As you can see below, two images about the same topic can be very different. A picture I took of my cat and one taken by a professional, have subtle yet important differences. If someone is simply searching for images of cats, the quality of the images we showcase make a big difference in whether a search session is successful. In this post we will explore the characteristics and techniques we employ to ensure that we are doing all we can to surface high quality images without compromising the relevance and context of your searches.
Example of high quality image:
Example of amateur image:
Understanding Image Quality
Image understanding is how we describe the overall approach that Bing employs to understand images. Below, we outline some of the different signals we use to determine whether a given image is high, medium or low quality.
For example with a query like “photos of yunnan”, where Yunnan is a beautiful resort in China, users expect to not only see relevant but also beautiful images that include the breathtaking scenery that define the region. Below you can see Bing’s image search results.
In contrast, Google’s results predominately showcase maps.
Understanding Objects
Historically, search engines have relied on information provided in the surrounding text of an image or on the corresponding web page to assess what is being captured in a given image. But with image understanding Bing can deduce what object is being captured within the image. We can isolate the object from the rest of the elements and derive its size and position. For instance, is the object in the foreground or in the background? How large is it in relation to the rest of the image? Below are three examples of our latest saliency map detection results. On the right side, you can see that the objects strike out from the rest of the image.
Understanding Color
Color is another important signal that Bing uses to determine high quality images. For instance, Bing can detect if a mass of color is associated with a particular object and then use that to determine if that is in contrast with the background (or vice versa.)
Here’s an example of color detection.
Filtering Image Characteristics
As you well know, beauty is in the eye of the beholder. One of the ways we help people narrow down their searches is by including filters. Using image understanding we let you filter your search by size, color, layout, date and even license (e.g. Creative Commons). So in a few short clicks you can filter for just black and white images.
Here is an example with the “black and white filter” turned on.
Faces, Shoulders and Bodies
Searches involving people are the most popular category of queries on Bing. Based on our relevancy data, we know that users are usually interested in pictures where people are readily recognizable. With this in mind, Bing is able to identify faces, shoulders and bodies as measures for high quality images of people – this is particularly important for celebrities.
Here the user has filtered by “faces.”
Here the user has filtered by “head and shoulders.”
Understanding Image Style
Styles of images are also important signals that we can use to determine visual quality. For instance, if you’re searching for cats because you’re looking to design a birthday card for your friend, you can narrow down your search by using the clip art filter. More importantly, the value of understanding the image styles is that when you search for celebrity or scenery photos, we don’t want to show clip art, line drawings or maps.
Here is an example with “clip art filter” turned on.
Here is an example with the “line drawing filter” turned on.
These and many other image content signals are used to train machine learning models to select the best quality images. When users issue a query to Bing, the top results are not only relevant but also have the best visual quality.
Better Image Quality by Smart Cropping
Today is a mobile first world. Ubiquitous tablets and phones introduce a new set of constraints due to their varied screen sizes. For instance, a good image experience for an iPhone could be vastly different than one for a Surface. Choosing the images of high visual quality is only part of the solution. We also need to be able to answer users’ questions by providing the best content in various screen sizes and UI configurations. The key question is how to preserve the best part of the image if we have to crop and resize to fit particular dimensions. To address this, we use a technique called smart cropping that preserves the “region of interest” for a given image.
Below is an example of how we use bounding boxes in order to provide flexible cropping results. When we show the images on smaller screens, we can still keep the region of interest to fit into the flexible UI configurations.
Human faces are naturally part of the region of interest in many cases. Our smart cropping is designed to keep the faces in the images after cropping. Below is an example when we fixed a user dissatisfaction of the image viewing experience. The faces were cropped off prior to smart cropping.
After we apply smart cropping to our image viewer, it brought in better visual quality.
Better Thumbnail Visual Quality
The same image source with the same cropping can still result in different visual quality. For the web images Bing crawled, we cache a smaller version of the image i.e. thumbnail to make the search faster. If the thumbnail visual quality is not as good, the user experience will not be perfect. Our optimization of thumbnails are across backend to frontend. It includes a backend advanced codec, and fine tuning the quality factors in every component all the way to the browsers.
If you compare our visual quality of images answers in both Bing and Google, you’ll notice a striking differences. To be fair, you can just look at the same image that both Bing and Google included, which is highlighted in the red box. Bing thumbnails are crisper while Google is blurry and loses details.
Here is screenshot before the effort of improving thumbnail visual quality. Our thumbnail visual quality has come a long way.
Building Hero Image Experiences
In cases where we have a large canvas with which to display high quality images, we show what we refer to as “Hero Images.” In the example below, you can see the larger image of the Giraffes is displayed in the upper left of the results. Based on retinal tracking observation in our R&D labs we know that people (in the west) generally scan from left to right so in cases where we have high confidence in an image we double the size and show it first.
Hero image example of {Giraffes}
Hero image example of {Jennifer Aniston}
It’s worth mentioning that Windows 8.1 smart search experience is powered by Bing. There are similar hero image experiences in it. By understanding the image color themes, we have applied the color themes to render a harmonious background color in Windows 8.1 Smart Search experience. Below you can see that the overall theme of the page is tuned to dominant violet of Katy Perry’s hair.
Example of colorization of Smart Search {San Francisco}
At Bing, understanding images and providing the best visual quality is one way we’re working to provide you with a great user experience. We’re just starting on this journey so please stay tuned for more to come.
– Dr. Eason Wang, Program Manager, Bing Multimedia Search
