When is a Chinese restaurant not a Chinese restaurant?

Think software testing is easy? Then you’re probably doing it poorly. Good software testing can be fun, but it’s never easy.

Try this thought experiment. When a user searches for “Chinese restaurant” or “burger joint” or even just the word “vegetarian”, Bing provides information on nearby restaurants that carry that type of food. As a tester, would you be able to verify that Bing handles these queries successfully?

• Perhaps you would type in “Chinese restaurant”, “Indian restaurant”, “Greek restaurant” and so on, to see what results show up. But that doesn’t scale. Bing supports two dozen different cuisines! And, since these results are location-specific, you would also need to check thousands of towns and zip codes. That would mean a lot of typing in queries and staring at results.
• You could outsource the task to vendors to solve the scaling problem, but that would escalate your costs. And does a vendor really understand your product well?
• You could decide not to test at all, and just wait to see what your users complain about. This solution is cheap in the short term, but do you really want your users to find your bugs for you? That’s a poor way to treat customers!

No, software testing is not easy. Test engineering and automation have changed tremendously in the past few years, and serious testers work hard to design innovative solutions. For instance, Bing currently combines human intelligence with computing power to deliver great software.

Here’s a hint of how Bing engineer René Barranco tested the restaurant queries above: 

1. Send Bing thousands of search strings such as “Chinese restaurant 98004” and “Barbeque near Peoria, Illinois”.

2. Write heuristics to evaluate the search results. A supposedly Chinese restaurant that doesn’t have words such as “Canton”, “Mandarin”, “Sichuan” or “Hunan” on its website, or lacks wontons on the menu is worth closer investigation.

3. Check that the restaurants are reasonably close to the target location. If one or more of the restaurants seems too far away, look more closely at that search result.

4. Analyze overall patterns in the answers. Are results more accurate in some regions? Are some cuisines handled better than others?

That’s just the start. Software testing isn’t easy, but it is challenging and rewarding when done well. We aim to do it really well.

Want to learn more? Check out Bing’s Software Design in Test job opportunities and join us in innovating how the world tests software  and services.

– Harry