Do not use

Bing's Transition to LLM/SLM Models: Optimizing Search with TensorRT-LLM

Bing Team — Tue, 17 Dec 2024 20:21:21 Z

At Bing, we are always pushing the boundaries of search technology. Leveraging both Large Language Models (LLMs) and Small Language Models (SLMs) marks a significant milestone in enhancing our search capabilities. While transformer models have served us well, the growing complexity of search queries necessitated more powerful models. LLMs can be expensive to serve and slow. To improve efficiency, we trained SLM models (~100x throughput improvement over LLM), which process and understand search queries more precisely.

Optimizing with TensorRT-LLM
One of the key challenges with larger models is managing latency and cost. To address this, we have integrated Nvidia TensorRT-LLM technique into our workflow to optimize our SLM inference performance.

One of the products where we leverage TensorRT-LLM is in ‘Deep search’. Deep search leverages SLMs in runtime to provide the best possible web results to Bing users.

This experience involves several steps, including understanding the user's query intent and ensuring the relevance and quality of web results. Given that SLMs require time to execute multiple steps, it is crucial to deliver value to users as quickly as possible. However, our product is built on the foundation of providing the best results, and we will not compromise on quality for speed. This is where TensorRT-LLM comes into play, reducing model inference time and, consequently, the end-to-end experience latency without sacrificing result quality.

TensorRT-LLM is a powerful optimization tool that helps us reduce the latency and cost associated with hosting and running large models on Nvidia A100 GPU.

Before optimization, our original Transformer model had a 95th percentile latency of 4.76 seconds per batch and a throughput of 4.2 queries per second per instance. Each batch consists of 20 queries. After integrating TensorRT-LLM, we achieved a 95th percentile latency reduction to 3.03 seconds per batch and increased throughput to 6.6 queries per second per instance. This optimization not only enhances the user experience by delivering quicker search results but also reduces the operational costs of running these large models by 57%.

	vLLM (V0.2.1) FP16 (P 50%/P 95%)	TensorRT-LLM (V0.9.0) int8 SmoothQuant (P 50%/P 95%)
Labeling	2.96/4.76	1.99/3.03

Optimization Technique:

TensorRT-LLM improves model performance:
The SmoothQuant technique was introduced in https://arxiv.org/abs/2211.10438. It is a method to run inference using INT8 for both activations and weights while maintaining the accuracy of the network (on downstream tasks).

As explained in the research paper, preprocessing must be applied to the weights of the model. TensorRT-LLM includes scripts to prepare the model to run using the SmoothQuant method.

Benefits for Users
The transition to SLM models and the integration of TensorRT-LLM bring several benefits to our users:

Faster Search Results: With optimized inference, users can enjoy quicker response times, making their search experience more seamless and efficient.
Improved Accuracy: The enhanced capabilities of SLM models allow us to deliver more accurate and contextualized search results, helping users find the information they need more effectively.
Cost Efficiency: By reducing the cost of hosting and running large models, we can continue to invest in further innovations and improvements, ensuring that Bing remains at the forefront of search technology.

Looking Ahead
As we continue to innovate and refine our search technology, we remain committed to providing the best possible experience for our users. The transition to LLM and SLM models and the integration of TensorRT LLM are just the beginning. We are excited about the future possibilities and look forward to sharing more advancements with you.

Stay tuned for more updates as we continue to push the boundaries of what's possible with search technology.

MSN Weather's Meteorological Makeover: Reimagined Weather Homepage

Bing Team — Fri, 22 Nov 2024 21:56:16 Z

MSN Weather is thrilled to announce the launch of its upgraded Desktop Weather Homepage! This update brings a fresh, modern design and an enhanced user experience aimed at providing you with accurate and intuitive weather information in a delightful and engaging way.

Key Updates:

1. Modern Design Style and UX
The new visual style aligns with the latest design trends, delivering an attractive and engaging experience. You'll enjoy a sleek interface that makes navigating weather information more enjoyable than ever.

2. One-Page Experience
With MSN Weather’s updated one-page design, you can access all the information you need in one convenient scroll. Key features like monthly forecasts, an updated hourly forecast experience, and a new trends and records section are now seamlessly integrated.

3. Introducing Weather Details
Explore our new “Current Conditions” section, featuring beautifully designed cards that provide real-time updates on weather parameters, including future peak times and helpful insights. Stay informed and engaged with the latest weather trends right at your fingertips!

The newly updated weather homepage highlights MSN Weather’s AI-enhanced weather forecasting capabilities that have become the staple for Microsoft’s global weather products. MSN Weather has been consistently recognized for its world leading forecast accuracy*. You can find weather information from MSN Weather through its integration into Windows 10, Windows 11, Microsoft Edge, Bing, and in the Bing and MSN mobile apps.

Author: Ting Sun, Principal Product Manager

*ForecastWatch, Microsoft_Report_2023 version 2 - April 05, 2024 (msn.com).

MSN Weather Presents: What's Up with Your Local Weather?

Bing Team — Tue, 17 Sep 2024 15:47:43 Z

Weather records are being shattered all around the globe, but what about where you live? Have you noticed any changes in your local weather?

“Is this heatwave just a fluke, or is it a sign of things to come?”

“It’s been so rainy this month! Is this normal?”

To help you stay informed, MSN Weather has introduced a new system that identifies unusual weather trends in your area and notifies you if they are significant.

The new Climate Insights Engine by MSN Weather leverages up to 70 years of detailed historical weather data to track temperature, precipitation, humidity, and wind across the globe. This engine compares your local weather to historical trends to determine how unusual recent weather has been. When something out of the ordinary happens, the system will notify you to keep you informed about your local climate.
Here are some examples of what the new engine can tell you:

"This month’s temperatures are 4.5 degrees lower than normal for your location.”
"It’s been 35 days since there has been measurable precipitation at your location, this is the 2nd longest streak ever recorded."
And... MSN Weather’s favorite new insight is referred to as the ‘Trendbuster’: "It’s been 65 days since the temperature has been above 11 degrees, which is a new record streak for your location, but it looks to break on Tuesday!"

The new Climate Insights Engine is the perfect companion to MSN Weather’s Weather Trends Page that debuted last winter empowering users to see monthly and yearly records, as well as tracking recent weather trends for their location. The insights engine will proactively inform you when significant trends and records have happened, while the trends page provides an attractive and useful tool for exploring deeper information about the climate patterns where you are.

MSN Weather’s new climate insights notification feature wouldn’t be possible without MSN Weather’s AI-enhanced weather forecasting capabilities that have become the staple for Microsoft’s global weather products. *MSN Weather has been consistently recognized for its world leading forecast accuracy. You can find weather information from MSN Weather through its integration into Windows 10, Windows 11, Microsoft Edge, Bing, and in the Bing and Microsoft Start mobile apps.

Authors: Matt Corey, Senior Product Manager, Alex Brant, Senior Software Engineer

*ForecastWatch, Microsoft_Report_2023 version 2 - April 05, 2024 (msn.com).

Improved Radar and Satellite Nowcasting for Clouds and Rain by Weather from Microsoft Start

Bing Team — Thu, 23 May 2024 15:35:36 Z

Microsoft Start’s new AI forecast model capabilities have unlocked the ability for users around the globe to experience high-quality, up-to-the-minute forecasts and maps of both cloud and precipitation together while also filling in gaps for data availability.

Since 2021, Weather from Microsoft Start has been running an operational short-term precipitation nowcasting model powered by generative AI to empower its users to make informed weather decisions. Every 2 minutes, this cutting-edge model provides users with forecasts at a hyper-local 1-kilometer resolution for up to four hours in the future. Since its initial presentation at NeurIPS 2021, the model has undergone continuous enhancements to improve precipitation forecast and map experiences across Microsoft’s weather products. In internal testing on benchmarks such as the SEVIR dataset, Microsoft Start’s model consistently ranks near the top while also providing forecasts up to two times further out compared to other generative AI models including DGMR (2021) and PreDiff (2023).

Traditionally, precipitation nowcasting models rely on weather radar data to “see” where precipitation is occurring and extrapolate how it will evolve. Deep learning models are capable of extracting information from very large volumes of data, and other data sources such as geostationary satellites, further provide vital information for precipitation forecasting. Using this data, Weather from Microsoft Start has developed a new AI model for Joint Global Cloud and precipitation nowcasting.

Adversarial regularization

Similar to the approach in DGMR, Microsoft used an adversarial learning approach, also known as a generative adversarial model (GAN) to improve the realism of model’s predictions. This approach introduced spatial and temporal discriminators to force the forecaster (generator) to produce high visual fidelity and temporal consistency. The spatial discriminator randomly samples forecast frames to improve visual fidelity, while the temporal discriminator samples chunks of several consecutive time frames for improving temporal consistency. During the training process, the generator tries to make predictions which look like real samples from the training data, while the discriminators try to distinguish between generated samples and real samples. Critical to this learning process was the introduction of skewed sampling favoring more frequent selection of frames at longer lead times (Figure 1), which helped reduce blurriness in forecasts further out, where typical regression loss favors overly smooth predictions.

Modifications to the loss function

The training loss function consists of pixel-wise regression loss and the adversarial loss (discriminator loss). The easiest way for the generator to fool the discriminators is by dissipating precipitation to zero, which resembles observed states of “no precipitation”. Since the discriminators are unable to distinguish between a prediction and truth in this scenario, the generator loss is designed to penalize for missing rain by introducing a recall control hyperparameter, α (Equation 1), which penalizes the model for negative bias in radar prediction. The α parameter is tuned by trading-off missed rain instances with an increased rain bias in test datasets.

Since the error in model predictions increases with lead-time, equally weighing these errors in the loss worsens shorter lead time forecasts. To counter this effect, Microsoft introduced a weighting ωt, which decreases with lead time, which results in both, i) an improvement in shorter lead-time forecasts, where the regression loss is important, and ii) better visual fidelity at longer lead time when the discriminator loss is more important. For pixel-wise loss we opt for L1 loss instead of L2, so that the model is not overly penalized for missing extreme precipitation conditions that may occur. Finally, a similar loss function with recall control and lead-time varying weight is applied to each of the outputs in Satellite + Radar Nowcasting.

Using both satellite and radar

Since late 2021, Microsoft Start’s has offered precipitation nowcasting globally, including in regions without radar coverage, thanks to geostationary satellite data providing near-global, high-resolution imagery of clouds and water vapor that can be used by AI models to deduce precipitation. This model provides simulated radar imagery to regions where radar is unavailable using satellites. Despite this achievement, the model performance was limited by availability of satellite imagery. Depending on the region, satellite imagery is only available about 85-95% of the time for acceptable latency.

With evidence suggesting the need for a separate decoder per task and a separate discriminator for each predicted channel, Weather from Microsoft Start built a model 4X bigger model than the previous one that only predicted simulated radar reflectivity. Finally, the new model jointly predicts both satellite and simulated radar reflectivity, enabling its predictions to fill data availability gaps. Since the precipitation task is more important than the satellite prediction task, the radar channel was given 6X more weight in the training loss function than satellite channels.

To evaluate the model performance, simulated radar reflectivity is evaluated by checking precision and recall for different reflectivity thresholds indicative of varying rainfall. Satellite image predictions were compared against persistence using metrics such as MSE, MAE, image quality metrics like PSNR, and MS-SSIM for similarity and FID scores for sharpness. Against the prior baseline of radar-only predictions, Microsoft Start’s new model presents a marked improvement in F1-score. Additionally, it was observed that predicted satellite images score better than a persistence forecast after 15 minutes, meaning these predictions can be used when satellite outages last longer than 15 minutes.

Considerations for operations

Productionizing a global forecast model with up-to-the-minute data presents its own challenges. A global inference is done using small sliding windows (tiles) with some overlap. This tile size is constrained by memory during model training, but not during inference. A small tile size during inference leads to high latency and bigger segmentation effects, to counter this, the generator architecture needs to meet three conditions: translation equivariance, spatially unconstrained operations, and low memory footprint of the hidden state. Consequently, Weather from Microsoft Start has developed its own unique video prediction model to meet these conditions, which allows flexibility in window sizing, thereby giving the ability to vary window size during training and inference.

This new model has unlocked the ability for users to experience seamless cloud and precipitation forecasts and maps while still providing accurate forecasts even when satellite data feeds experience unexpected outages. The new Satellite + Radar nowcasting model is the latest addition to Weather from Microsoft Start’s growing inventory of world-leading weather models. According to an independent study commissioned by Microsoft, *Weather from Microsoft Start was recognized for its leading forecast accuracy. You can find weather information from Weather from Microsoft Start through its integration into Windows 10, Windows 11, Microsoft Edge, Bing, and in the Bing and Microsoft Start mobile apps.

*ForecastWatch, Analysis of One-to Five-Day-Out Global Temperature, Wind Speed, Precipitation and Opacity Forecasts, Jan-Jun 2022 (msn.com).

Weather from Microsoft Start’s new AI capabilities are improving 30-day weather forecasts

Bing Team — Fri, 03 May 2024 16:26:21 Z

In a newly published article on ArXiv, the research team at Weather from Microsoft Start has shown how AI weather models compare to the latest state-of-the-art European Centre for Medium-range Weather Forecasts (ECMWF) extended-range ensemble. Instead of using a single type of AI model, we combine five different trained models comprised of three different deep learning architectures together to produce some promising forecasts one month in advance.

In 1972, Edward Norton Lorenz, one of the pioneers of numerical weather prediction (NWP), famously stated that “a butterfly flapping its wings in Brazil can produce a tornado in Texas.” This vivid metaphor was intended to demonstrate the chaotic nature of the atmosphere, where even the tiniest influence can result in a wildly unpredictable outcome. Scientific research has suggested that even with perfect weather models and nearly perfect data, it becomes very difficult to predict phenomena such as thunderstorms even one or two days ahead.

So how, then, can we hope to make useful weather forecasts all the way out to 30 days? Unsurprisingly, if we look at a single simulation by an NWP model, this forecast would be wildly inaccurate most of the time. However, decades of scientific research in ensemble forecasting have shown that it is possible to tease out information in long-range forecasts by relying on probabilistic forecasts – running dozens or even thousands of different but equally-likely simulations of the weather and extracting meaningful information from them.

NWP ensembles, such as the state-of-the-art system run by ECMWF, require large amounts of supercomputing resources and produce petabytes of data. However, recent advances in AI research have shown that deep learning methods can predict the weather much faster and even more accurately than traditional NWP models.

Unlike traditional models, which compute the evolution of weather around the globe by using physics of fluid dynamics in addition to approximations of other physical processes such as thunderstorms and wind turbulence, AI-powered weather prediction models learn from decades of observed weather to recognize patterns and predict their future evolution. They operate in much the same way as an NWP model, though: given the current state of the atmosphere on a 3-D globe (latitude, longitude, and height), predict the state of the atmosphere for some future time, say one hour later. They then feed this prediction back into the model to predict two hours later, and so on. Because the models can operate at much coarser spatial resolution and take much larger time steps than an equivalent thermodynamic model could, simulations take only minutes on a single graphics processing unit (GPU). Hence these models can run more frequently to produce more simulations for better probabilistic forecasts.

In our preprint, we compare our AI weather models to the state-of-the-art ECMWF extended-range ensemble, which makes forecasts at 0.4° spatial resolution every six hours up to 46 days ahead. The ECMWF model was last updated in June 2023 with an increase in ensemble size from 50 members to 100. Each of our five AI models was run 20 times to create an ensemble of 100 forecasts at 1° resolution in latitude and longitude every six hours into the future.

The results are quite encouraging: when measuring temperature errors using the Continuous Ranked Probability Score (CRPS) metric, our out-of-the-box AI ensemble outperforms the ECMWF model by 17% for one-week forecasts and 4% for four-week forecasts (Figure 1). The CRPS is optimized when the distribution of the ensemble matches the expected distribution of the observations, hence the model must correctly represent the uncertainty in a forecast. It can be thought of like a mean absolute error, where lower is better.

Figure 1: Temperature forecast error (CRPS; lower is better) for Microsoft’s AI ensemble and the ECMWF ensemble, for each week of forecast lead time.

The longer a model runs into the future, the more it tends to accumulate errors due to model drift biases. When running an operational model, it’s important to correct these systematic errors by learning from simulated forecasts of the past, or hindcasts, how the model tends to drift. When applying a correction, we observe that our AI ensemble scores fall behind the ECMWF ensemble’s by about 3% at week four.

We also consider what happens when combining the two ensembles together into a 200-member probabilistic forecast. It turns out that the result is better than either individual model, albeit by a very small (not significant) margin. This suggests that the AI ensemble is creating new variability in the forecasts that can help capture more weather phenomena such as extreme temperatures or precipitation, yet at the same time traditional forecasting methods remain useful. As we can see from the spatial distribution of forecast errors in Figure 2, which are very similar for our AI ensemble and the ECMWF ensemble, the predictability of each location’s weather remains the dominant factor in determining forecast accuracy rather than the specific model used for the forecast.

Figure 2: Spatial distribution of temperature forecast errors at week 4 (CRPS, lower is better).

As shown by our results, AI weather models have the potential to bring the next big improvements to weather forecasting beyond ten days. These 30-day forecasts will be the latest addition to Microsoft’s growing inventory of world-leading weather modeling. According to an independent study commissioned by Microsoft,* Weather from Microsoft Start was recognized for its leading forecast accuracy. You can find weather information from Weather from Microsoft Start through its integration into Windows 10, Windows 11, Microsoft Edge, Bing, and in the Bing and Microsoft Start mobile apps.

*ForecastWatch, Analysis of One-to Five-Day-Out Global Temperature, Wind Speed, Precipitation and Opacity Forecasts, Jan-Jun 2022 (msn.com).

How are climate changes affecting your location? Find out with Weather from Microsoft Start.

Bing Team — Thu, 22 Feb 2024 11:12:06 Z

Extreme weather seems to be in the news a lot lately. Weather events like stronger tropical cyclones, record rainfalls, extended droughts, heat waves, wildfires and smoke are all breaking records more frequently, impacting the lives of millions.

In fact, the past 8 years are the warmest ever recorded – as far back as 1850 – and 2023 was the warmest year in NOAA’s 174 year record. This leaves more people wondering, “How are climate changes affecting me and my local weather?“ Weather from Microsoft Start recently launched a new feature to help people find answers to questions like these.

The new Weather Trends page is a powerful feature which draws on up to 70 years of global weather history. It can help users learn how their recent weather compares with the past by year, month, or even by day. Historical averages and detailed records show how many rainy, snowy, cloudy, or sunny days there typically are this month, and visualization tools such as innovative new pie charts make it easier to understand and compare the data.

The true value of the new Weather Trends page is that all of this powerful information is tuned for actual locations, not lumped into a regional or national weather story that may have limited relevance.

In addition to the Weather Trends page, Microsoft is now testing its new Climate Insights Engine. This engine smartly leverages the Earth’s climate history and Microsoft’s AI intelligence to identify and notify users of impactful weather trends at their specific location.

According to an independent study commissioned by Microsoft,* Weather from Microsoft Start has already been recognized for its world leading forecast accuracy, offering critical weather information to millions of daily users. It’s deeply integrated into Windows 10 & 11, powering the weather alerts you may have seen on your taskbar. It’s also integrated into Edge, Bing, and available on the Bing and Microsoft Start mobile apps.

With the launch of the new Weather Trends page and the soon to be launched Climate Insights Engine as additions to the most accurate weather forecast in the world,* Microsoft is using AI to empower people with the weather information they need to stay safe.

*ForecastWatch, Analysis of One-to Five-Day-Out Global Temperature, Wind Speed, Precipitation and Opacity Forecasts, Jan-Jun 2022 (msn.com).

Introducing deep search

Bing Team — Tue, 05 Dec 2023 05:51:40 Z

Today’s search engines are powerful tools that help us find information on the web, but sometimes they fall short of our expectations. When we have complex, nuanced, or specific questions, we often struggle to find the answers we need. We ourselves know what we’re looking for, but the search engine just doesn’t seem to understand.

That's why we created deep search, a new Microsoft Bing feature that provides even more relevant and comprehensive answers to the most complex search queries. Deep search is not a replacement for Bing's existing web search, but an enhancement that offers the option for a deeper and richer exploration of the web.

Understanding search intent

Deep search builds on Bing's existing web index and ranking system and enhances them with GPT-4. GPT-4 is a state-of-the-art generative AI LLM (Large Language Model) that can create natural language text from any input. In the case of deep search, GPT-4 takes the search query and expands it into a more comprehensive description of what an ideal set of results should include.

For example, let’s say I’m researching loyalty programs in different countries and search for "how do points systems work in Japan". Deep search might generate a more comprehensive description like this:

Provide an explanation of how various loyalty card programs work in Japan, including the benefits, requirements, and limitations of each. Include examples of popular loyalty cards from different categories, such as convenience stores, supermarkets, and restaurants. Show a comparison of the advantages and disadvantages of using loyalty cards versus other payment methods in Japan, including current rewards and benefits. Highlight the most popular services and participating merchants.

This expanded description captures my intent and expectations more accurately and clearly than a few keywords. It also helps Bing understand what kind of information I am looking for.

Some queries are ambiguous. For example, "how do points systems work in Japan" could refer to rewards points, as I intended, but could also be seeking information on immigration policy or something else. The expanded description may be right for one of those interpretations but not for all of them. In that case, deep search leverages GPT-4 to find all the possible intents and computes an comprehensive description for each of them. Deep search offers a disambiguation pane where all of these intents are represented. If my research intent was misunderstood, I can select the right one from the disambiguation pane, and the corresponding comprehensive description will be used instead.

Finding deeper results

Bing then goes much deeper into the web with that task in mind, pulling back relevant results that often don't show up in typical search results. Deep search uses a combination of querying techniques to find pages that might match my expanded query, rewriting the query on my behalf, and searching for those variations too.

For example, for my earlier loyalty points query, deep search might also try searching for:

loyalty card programs Japan
best loyalty cards for travelers in Japan
comparison of loyalty programs by category Japan
redeeming loyalty cards in Japan
managing loyalty points with phone apps

By doing this, deep search can find results that cover different aspects of my query, even if they don't explicitly include the original keywords. Regular searches on Bing already consider millions of web pages for each search and deep search does ten times that to find results that are more informative and specific than the ones that rank higher in normal search.

Ranking results

Once deep search has gathered a wide collection of web pages to review, it then ranks them according to how well they match the comprehensive description. Deep search uses a variety of signals to determine the relevance and quality of each result, considering factors like how well the topic matches, whether it’s at the appropriate level of detail, how credible and trustworthy the source is, how fresh and popular it is, and so on.

By doing this, deep search can present a curated list of results and answers that are more likely to answer your question, satisfy your curiosity, or solve your problem.

Speed

Going deeper takes time, and deep search can take up to thirty seconds to complete. This might seem like a long time compared to normal search, but it can be worth the wait for more specific or comprehensive answers.

Deep search is not meant for every query or every user. It's designed for those who have complex questions that require more than a simple answer. Bing will always return regular search results in less than a second and deep search is an optional feature.

More ways that GPT-4 is used in Bing

GPT-4 is already used in many places on Bing, including Copilot, Image Creator from Designer, and already in regular web result ranking where, in January, it powered one of the single biggest relevance improvements in Bing’s history.

We’re excited for the potential for deep search to offer yet another significant improvement in search result quality. This is currently an experimental feature that is available to randomly selected small groups of users on Bing worldwide while we are testing and improving it. We would love to hear your feedback and suggestions on how to make deep search better.

Continued AI Innovation in Copilot

Bing Team — Tue, 05 Dec 2023 05:50:59 Z

Today, in addition to celebrating the first year of Microsoft Copilot, we announced several new Copilot features that are beginning to roll out. We’re excited to give you a more detailed glimpse into some of these features.

GPT-4 Turbo – Soon, Copilot will be able to generate responses using OpenAI's latest model, GPT-4 Turbo, enabling you to tackle more complex and longer tasks such as writing code and more. This model is currently in testing with select users and will be widely integrated into Copilot in the coming weeks.

New DALL-E 3 Model – You can now use Copilot to create images that are even higher quality and more accurate to the prompt with an updated DALL-E 3 model. These capabilities are available to you now by visiting bing.com/create or by prompting Copilot to create an image.

See the below comparison, although to truly appreciate the level of detail of the new model click on the prompt here: photorealistic stegosaurus having its bony plates attended to by a nail salon

Multi-Modal with Search Grounding – We are combining the power of GPT-4 with vision with Bing image search and web search data to deliver better image understanding for your queries. This new capability will be available soon.

Here you can see how we’ve updated Prometheus for multi-modal:

The results are pretty impressive as you can see in the example below. A traditional multi-modal system would be able to describe what's in the image in a generic way, but with search grounding we are able to identify the exact shuttle and the day it was launched.

Code Interpreter - We are developing a new capability that will enable you to perform complex tasks such as more accurate calculation, coding, data analysis, visualization, math and more. We are gathering feedback on these capabilities from a select set of users and plan to make it widely available soon.

Copilot will write the code to answer your complex, natural-language requests, run that code in a sandboxed environment and use the results to give you higher quality responses. You can also upload and download files to and from Copilot, so you can work with your own data and code as well as Bing search results.

Copilot's powerful Python environment runs in a secure sandboxed environment built on top of Azure Container Apps. It provides fast, user isolated environments preloaded with many popular tools and libraries for data science, such as pandas, numpy, matplotlib, sklearn, flask, and more to solve complex problems. You can combine the data Copilot has access to from Bing search and the web with data uploaded in files to get deep, relevant insights as well as beautiful and interactive outputs.

Video understanding and Q&A – Copilot in Edge. You can now summarize or ask questions about a video that you are watching in Edge. For example if you are watching the YouTube video of Satya’s latest Ignite keynote, you can ask Copilot to summarize it for you as you can see in the screenshot below.

We look forward to your feedback!

Turing Bletchley v3 - A Vision-Language Foundation Model

Bing Team — Tue, 29 Aug 2023 12:02:43 Z

Today, the Turing team is excited to go into detail on how Turing Bletchley v3, a new multilingual vision-language foundation model, is used across Microsoft products. This model is the latest version of Turing Bletchley – our series of multilingual multimodal foundation models which understand more than 90 languages. About two years ago we introduced Turing Bletchley v1 to the world via our blog post. Turing Bletchley v1 was among the first multilingual vision-language models – importantly these allow performant image search in both popular and low-resource languages. Since then, we have iterated on datasets, modeling, and pretraining to improve all aspects of our model, with the end goal of providing better search capabilities across languages. Last fall we released Turing Bletchley v3 internally, and it has now shipped in multiple products across Microsoft. You might already be using Turing Bletchley v3 when searching in Bing, and in this post, we will share more details regarding the model which powers the next-generation of multimodal and multilingual scenarios at Microsoft.

Turing Bletchley v3 enables performant image search across more than 90 languages

What is a vision-language model?

Humans use multiple senses for understanding the world – vision, hearing, taste, and so on. Vision language models are models which have two such “senses” – vision and language. More concretely, the model can take as input both images and text. Turing Bletchley v3 is a vision-language model which takes as input either an image or a snippet of text and outputs a vector – a location in some high-dimensional space. The goal is to have the locations corresponding to an image I and a text snippet J be “close” to each other if the image I and text J are semantically related. For example, we would want the locations described by the text snippet “a cat in a field of grass” to be “close” to images of cats in grass. These types of models can be used for example in image search – the vector of the user query is simply compared to the vector of all images in your database.

A vision-language model like Turing Bletchley can take either images or text and embed them as points in a vector space. Embedded sentence like “Mount Fuji” or “富士山” (Mount Fuji in Japanese) will be close to embedded images Mount Fuji in this vector space.

Pretraining

One technique that significantly improves Turing Bletchley v3 compared to v1 is the use of masked multimodal learning. This is a new training paradigm developed collaboratively by the Turing team and MSRA to empower efficient multimodal learning, and the paper describing this method was recently presented at CVPR 2023. Given an image and a caption describing the image, some words in the caption are masked. A neural network is then trained to predict the hidden words conditioned on both the image and the text. The task can also be flipped to mask out pixels instead of words. This type of masked training together with a large transformer-based model leads to a strong pre-trained model which can be finetuned on a diverse set of downstream tasks. When evaluated on several English-only vision language tasks this strategy reaches a new SOTA in many of them. We improved the SOTA for the NLVR vision-language reasoning task by more than 5 points and achieved a new SOTA for object detection on COCO – the first time a pure vision-transformer model topped the leaderboard.

Masked pretraining improves upon the state-of-the-art (SOTA) vision language models on multiple tasks with a significant margin.

Most Microsoft products are served worldwide, e.g., Bing has millions of users spread over more than 100 countries. To serve this diverse demographic, Turing Bletchley v3 needs to be multilingual. The production model is thus trained in two stages on multilingual data. First a mask-based training stage using both image-caption pairs and text pairs where the same sentence is given in two different languages. Then a contrastive training stage which also uses translated language pairs. The resulting Turing Bletchley v3 model has strong zero-shot image and text capabilities in up to 94 languages, where we have balanced the performance of public academic benchmarks and internal production benchmarks. There are multiple evaluation benchmarks for multilingual and multimodal models, we have used an internal version of WIT – which relies on images and captions from Wikipedia. The WIT dataset contains many languages and allows us to assess the performance of Turing Bletchley across many low-resource languages. As can be seen, Turing Bletchley v3 gives strong improvements for both image and text retrieval. V3 gave significant improvements over v2, and hence we skipped productionizing Turing Bletchley v2.

Evaluation on the WIT dataset from Wikipedia. We stratify the scores by image retrieval (IR) and text retrieval (TR) for the top 25 and top 50 languages. Turing Bletchley v3 shows strong improvements on multilingual retrieval over Turing Bletchley v1.

Images for Question-Answering

Question-and-answering is a feature in Bing where the user gets a direct answer in addition to links to external websites. In addition to showing text in this section, we show relevant images to delight our users. When judged by humans, Turing Bletchley v3 improves an internal quality metric that measures the relevance of the images to the query and answer pairs by 9 points for this scenario.

Two examples of images retrieved when answering questions on Bing. Both images are closely associated with the user query and the displayed passage: showing a panda eating and a scenic view of Seattle’s space needle.

Image Search in Bing

A very natural application of vision-language models is image search, i.e., given a user query or image, retrieve the most relevant image to show to the user. A distilled version of Bletchley v3 can be used in two such image search scenarios in Bing – image-to-image search and text-to-image search. 1) In the former case the user might have access to an image, and Bing retrieves similar and relevant images. After using the new vision-language model, user engagement (measured via clicks) increases by more than 6%. 2) In the latter case the user inputs a short text query and expects relevant images to be shown. Here we see an improvement of about 0.76 Discounted Cumulative Gain (DCG), a common metric used for measuring retrieval quality.

Example of image-to-image search. Given the image to the left, depicting dry and cracked earth, we can retrieve and show the images to the right. They all contain dry and cracked earth.

Content moderation for Xbox

Xbox uses automation to support player safety. The Xbox platform encourages players to be themselves and personalize their profiles, which includes uploading photos and images to their profiles; however, these photos and images cannot come at the expense of other players’ positive experiences. To ensure that only appropriate content is shown to the player base, the Xbox team implements various content moderation strategies including both human and automated methods. One such method is automatically identifying inappropriate images with machine learning models. Turing Bletchley is powering the latest version of the Xbox content moderation system, using its world knowledge to understand the many nuances for what images are acceptable based on the Community Standards on the Xbox platform.

Building the New Bing: Image Creator

Bing Team — Thu, 23 Mar 2023 17:21:47 Z

As we announced earlier this week, with Bing Image Creator you can now turn words into pictures to truly express your imagination, giving you access to infinite image possibilities right from within Bing. A vivid dream, a birthday invitation to fit a unique theme, the new concept you’re trying to land at work. These are personal and specific. These are things that require a unique image to express your unique idea. Even better is that you can ask for these pictures in the flow of your conversation with the new Bing chat.

Much like in my first blog about Building the New Bing, I’d like to share some of the thinking and the tech behind this magic, while providing some tips for how to best use the feature in the product.

Image Creator uses OpenAI’s latest DALL·E model, a powerful language model that can generate coherent and diverse texts on almost any topic. Except, instead of taking text as input and producing a text as output, DALL·E takes a text as input and produces an image as output. It uses a technique called self-attention to learn the relationships between text and images.

The results are realistic images that can combine concepts, attributes, and styles. For example, you can ask Bing Image Creator to draw a “cat wearing a hat and sunglasses” or a “surreal painting of a pineapple” and it will generate images that match your request. You can include a significant number of details in your query like in the image below. For the best results, you’ll want to include a subject, details and the scene.

Prompt: create a realistic photo of a distinguished rhino in a 3 piece suit drinking a cup of tea

Integrating the new Bing with the power of Image Creator further unlocks the potential of both of these tools. I find that it’s more intuitive and fun to create images iteratively within a Bing chat session, rather than directly in the Image Creator vertical. Plus, there are scenarios where it is so helpful to be able to create within the context of a chat session, for instance creating a character for the bedtime story you just wrote.

As you may recall, we announced Image Creator availability in a few markets back in October. So why did we wait so long to roll it out more broadly? Well, there are a few reasons:

We wanted to test and learn, especially regarding the implications for responsible AI. It’s important to note that in addition to generating AI images with the latest DALL·E models from OpenAI, we prohibit the use of Image Creator to produce content that can inflict harm on individuals or society. To achieve this, we block certain prompts and images, and we’ve used the last few months to help inform this approach. I realize this may result in over blocking some images sometimes which can be frustrating, and that’s why we’ve implemented an appeal process and will continue to improve our classifiers.
We were expecting an improved model from OpenAI. Their latest model has dramatically increased the quality of the images with more detail and understands more visual concepts, which leads to more relevant results.
Since last October, we have announced the new Bing, and it immediately became clear how intuitively Image Creator could fit into the chat experience. We were able to enhance the Prometheus model by providing instructions and examples of how to interact with Image Creator in a multi-turn fashion.

Of course, even with all of these adjustments, we are still learning and iterating so please keep providing your valuable feedback.

I’m delighted that Image Creator is now available to try for free in English everywhere that the new Bing is available. Here’s how:

For those in the new Bing preview:

Go to bing.com and go to the chat experience, and select the Creative conversation style.
Ask Bing to create an image for you. The more descriptive the better. For instance "create an image of an astronaut riding a skateboard, digital art" will create better images than “create an image of an astronaut.” Remember to type the query only in English at this point.
Every query gives you up to 4 images to choose from.
You can download or share your images and encourage others to give it a try.

If you’re not yet in the preview of the new Bing, you can still access Bing Image Creator by going to bing.com/create.

And if you’re not already signed up, here’s another reason to start using Microsoft Rewards. As a Microsoft Rewards member you can further accelerate your creative power by using points to boost the speed at which your image creations are generated.

As someone who speaks daily in my non-native language, I’m very familiar with the challenges of trying to fully articulate myself. It’s the same feeling you get when you can’t find a picture that captures exactly what you want to express. I think that’s one of the reasons why I’m captivated by the potential of Image Creator and how it empowers us to become our own visual storytellers.

Even though the internet is vast, I bet you can think of a time when you couldn’t quite find the right image. In a way, Image Creator infinitely expands the Bing image index since, if you can’t find an existing image, you can create it. I can’t wait to see what you all create with Bing Image Creator. We are only at the very beginning of our journey, and I'm excited to be a part of this new era in search in partnership with all of you.

Jordi Ribas
Corporate Vice President, Search & AI