Starting this week, we are sending out a number of new Crawl Error Alerts to users of Bing Webmaster Tools. These Crawl Error Alerts inform you about issues that we encountered during recent crawls of your site. You can find detailed descriptions and explanations for each of these alerts in the List of Crawl Error Alerts topic in our new Webmaster Help Center, but I’d like to give you a quick rundown of the new alerts and issue types and why you should take heed:
New Issue Type 1: DNS Failures
The first new alert deals with DNS issues we encountered when crawling your site. DNS is short for Domain Name System, which is the system that is responsible for resolving hostnames to corresponding IP addresses. When we request a page from your server and the address of your hostname does not get resolved, we are not able to fetch the page. This may indicate that the DNS service used for your website may be down or misconfigured. For example, the Time-To-Live (TTL) setting for the DNS record may have been set to 0, which can lead to problems. In some cases, the DNS service used for your website may be actively blocking or filtering Bingbot – something you may be able to check using the Fetch as Bingbot tool. In any case, we cannot crawl your site if this step fails and you may need to follow up with your DNS admin to see what’s going on.
DNS issues are mostly site-wide and not necessarily specific to the individual pages, but we will still report the number of attempts we made to fetch specific URLs and how many of those attempts failed. Not long after receiving the alert we will upload a sample set of URLs we tried to fetch when we encountered the DNS issue to the Crawl Information tool, were they can be found in a brand new column called DNS Issues.
New Issue Type 2: Connection Timeouts
The second new issue type we have added informs you about connection issues we encountered. This category of alerts highlights instances during which your web server refused a connection and or did not respond to our requests with a valid HTTP status in a timely fashion. Connection issues could indicate that your server was temporarily down or otherwise unable to respond. In some cases, it could mean that your server is actively refusing connections for Bingbot (something you may not necessarily be aware of!). In either case, some more investigation is needed. Same as for DNS issues, we will upload a sample of URLs that we couldn’t fetch due to Connection Errors to the Crawl Information tool where they can be found under the new Connection Timeouts column as shown in this screen capture:
Detailed 4xx and 5xx Server Response Alerts
In addition to the new issue categories and alerts concerning DNS and Connection issues, we are also sending a number of granular alerts when your server responded to requests from Bingbot with HTTP status codes in the ranges 400-499 and 500-599. Again, check out the List of Crawl Alerts for the full details, but know that we will only trigger alerts if we see a both a significantly large percentage as well as substantial increase in the amount of errors when compared to prior crawl cycles, so that you are made aware of potential spiking issues early on.
Here’s an example of an alert concerning a spike in 403 (Forbidden) responses, which indicates that your server may be unintentionally blocking valid Bingbot requests:
Anatomy of a Crawl Error Alert
As shown in the example above, each message contains a description of the problem, the number of times we saw the error type, as well as the percentage of all requests for which we saw this error during the most recent 24-hour crawl cycle. In addition, the message contains a direct link to the relevant section in the Crawl Information tool (when a corresponding category exists) where you will find a sample set of URLs for which we encountered the issue. Note that we upload these sample URLs after sending out all of our alerts (we want webmasters to know about spiking issues as soon as possible), so there may be a delay between you receiving the alert and us showing the additional data in the Crawl Information tool, but the sample URLs may help with any further investigation.
Stay on Top of Your Alerts and Other Messages
As a rule, all alerts go directly to the Message Center inside your Bing Webmaster Tools account and high-priority alerts also show up at the top of your site dashboard. However, to stay on top of new, unread messages in your Message Center outside of the Webmaster Tools — including the new crawl alerts, but also index issues, and other critical notifications such as malware warnings — make sure to set up your Webmaster Tools profile to receive digests of your unread messages. You can do this by following the steps described in the help topic How to Set Email Alert Preferences. Since alerts are more often the only way to find out about issues that can affect your performance in Bing (and as a consequence, Yahoo!), I strongly recommend you set your frequency to "daily".
We hope you find these new crawl error alerts useful and in helping you stay informed about your site’s health and ensuring that problems we encounter don’t go unnoticed.
Will you be you keeping an eye on your crawl error alerts? Have you set up your communication preferences to receive daily digests yet? Are there additional notifications you would like to receive for your site? Leave me a note in the comments!
Vincent Wehren –
Senior Program Manager, Bing Webmaster Tools