The world is a chaotic place. From the billions of Facebook updates made each day to the constant tick of stock quotes for the 3,571 companies listed on NASDAQ, there are trillions of pieces of information about the real world swirling out there every day. While there are plenty of apps and services that can help us manage what’s happening, we’re often left feeling buried by the weight of it all. Instead of feeling empowered, we feel overwhelmed.
At Bing, we believe that the technologies that power search are uniquely positioned to provide you with a more natural and personal approach to this problem. By now, you’ve heard about the upcoming release of Windows Phone including the introduction of the personal assistant, Cortana, which is powered by Bing. You can read about this in more detail in Joe Belfiore’s post here.
The Bing Platform: More Personal, Helpful and Intelligent
This post will explore the technology behind Cortana, Bing.com and other Microsoft products that is making all of this possible, a platform that combines Bing’s massive investments towards indexing the real world, with speech recognition, language and entity understanding, and stream processing that, taken together, make Bing more personal, helpful and intelligent. Instead of you having to monitor the 50 thousand flights in the air today, spiking news coverage of your favorite sports team or the 70 thousand cars on the interstate blocking your commute, the Bing platform processes these events in near real-time, and delivers to you information you need at the right time.
When you first launch Cortana from your Windows Phone, she takes time to get to know you by asking basic questions to better understand your needs including your name, your food preferences, what category of movie you like and more. Over time, Bing can observe, learn, and when confident make helpful suggestions to you.
Let’s look at my upcoming trip to Frankfurt as an example of how this works. The airline has sent me a confirmation email including my itinerary, flight number (Flight DL 0410), embarkation city (New York) and destination city (Frankfurt.) These items are recognized as structured data and are transmitted to the agent system within Bing which recognizes them as potential actions. Bing notes the locations, flight number, time and date and passes along a prompt to Windows Phone asking if I would like to be reminded of the flight. Fast forward a week. The stream processing system in Bing is proactively monitoring the status of Flight DL 0410 which thankfully is on time. Two hours before my flight Bing sends me a notification on my Windows Phone including real-time status of not only my flight but traffic updates and directions to JFK along with a venue map of the airport.
What’s going on within Bing when I tell my Windows Phone to remind me to take out the garbage when I get home? Let’s take a look.
In a matter of seconds, speech goes to the cloud where the individual words are translated into bits which are passed to the speech recognition engine. Using a system trained on deep neural networks (DNNs), the voice stream is converted into a sequence whereby natural language processing can infer that I’m asking a question. DNNs are a technology inspired by the functioning of neurons in the brain. In a similar way, DNN technology can detect patterns akin to the way people understand speech. From there, the instructions are passed along to the system which determines my intent. In this case, after having confirmed with me during a previous interaction the system knows that I’m referring to a specific address when I say “home.” From there, the system constructs a response. The Bing platform then implements a rule for the system to follow. For instance here, it is setting a geo-fence on the phone so that the reminder pops up when I reach home.
Here’s another example. What happens when I ask a question like, will it be chilly in San Francisco, tomorrow?
As in the previous example my speech is translated and processed by the system. While you and I know that by “chilly” that I’m asking about the weather but it’s not so simple for computers. Here we employ sophisticated natural language processing to correlate “chilly” to “weather.” The system then taps Bing’s knowledge repository which tracks billions of people, places and things in the real-world. The repository “knows” that San Francisco is a place in California. From there, Bing looks at its corresponding data feeds for weather throughout the United States to determine it will be sunny and passes that on to my Windows Phone which displays the forecast for San Francisco tomorrow.
Building on the work we have done to enhance experiences across a range of devices including Xbox, Office and Windows 8, Windows Phone 8.1 is the latest manifestation of what’s possible with the platform. Bing also shows up in places you might not expect, like Facebook, Yahoo, Siri on the latest iPhones and iPads, and even Android devices like the Amazon Kindle Fire.
A More Personal Bing.com
While a digital assistant is a bright example of how this comes together, you can also experience this personalized and helpful approach on Bing.com. In the coming days, we are unveiling a set of personalized cards on the Bing homepage that capture things that matter to you.
Now Bing can automatically help you keep track of the interests you care about. Just tag a topic in your Bing settingsand we will do the rest. Tag things you care about, and start seeing personalized news, weather, flights, stocks with more to come – all as part of the search experience you already love.
These new homepage tiles are just the start of a new batch of personalization features that you’ll increasingly see across Microsoft devices and services. You can read more here.
– Mike Calcagno, Partner Development Manager, Bing