Robots speaking many languages

We’ve already covered in past blog articles some of the basics about how webmasters can use a file called robots.txt to control how search engine crawlers (aka bots) crawl their websites. But there is so much more to talk about with bots. So let’s take a bit of a deeper dive into the subject. Topic 1: Using the proper text file encoding The robots.txt file is used by webmasters to either specifically define which files and directories...
Read More

MSNBot 1.1 is retired

The Bing team has been talking about its new crawler (aka bot), MSNBot 2.0b, in this blog for quite some time now. We have made numerous improvements in its performance, addressed some webmaster concerns, and published detailed information on how to control the bot with a robots.txt file. Today we are announcing that the new bot is fully operational. This development will enable Bing to do a better job at gathering the information we need from the...
Read More

Webmaster Center blog Q&A

We’ve been really busy here at the Bing Webmaster Center blog team, pumping out new content on a regular basis to create a nice library of content on issues that matter to webmasters and online publishers. I thought I’d take a moment to catch my breath, pause on creating a new thematic article (or yet another multi-part series!) for SEM 101, and address some commonly asked questions in the blog comments. Q: Why wasn’t my...
Read More

How Microsoft handles bots clicking on ads

There’s been some recent discussion in the SEO blogosphere asserting that Bing clicks its own adCenter ads. This has created some misunderstanding. Let’s take a moment to clarify what is actually happening, and what this really means for webmasters and advertisers. The Bing team is aware of an issue shared by all search engines: paid advertising links on sites are, on occasion, crawled and indexed by search engines. Standard practice...
Read More

Prevent a bot from getting “lost in space” (SEM 101)

We recently published a non-SEM 101 blog post on controlling the crawl rate of MSNBot, the Bing web crawler (aka robot, or simply just bot). That got me thinking about robots. Naturally, that led to The Robot on Lost in Space. Will Robinson, the show’s precocious youngster who was a whiz at 1960s-style, clunky electronics (even though the show was supposedly set in 1997!), was best friends with The Robot. They looked out for each other and...
Read More

Uncovering web-based treasure with Sitemaps (SEM 101)

Have you ever noticed how pirate treasure maps are like Sitemaps? While your website may not contain a treasure of gold and silver (unless it’s a metals commodities trading site!), if you have good content, that is certainly treasure to someone who is looking for it. Unfortunately, it’s buried on your website and no one knows what’s there except you! But since you want to share your site’s treasure with others, you need to...
Read More

Crawl delay and the Bing crawler, MSNBot

Search engines, such as Bing, need to regularly crawl websites not only  to  index new content, but also to check for content changes and removed content. Bing offers webmasters the ability to slow down the crawl rate to accommodate web server load issues. The use of such a setting is not always needed nor is it generally recommended, but it is available for use by webmasters should the need arise. Websites that are small (page-wise) and...
Read More

Common errors that can tank a site (SEM 101)

Imagine being a content developer for a website. You write a bunch of clever and informative articles, which should deliver a good dose of new visitors and ranking potential to the site. You submit them to the IT department for publishing online, and wait for good things to happen. But instead, it all falls flat. A look at your web analytics tools reveals that the number of site visitors has not increased over the time your new material was...
Read More

Heads up on <head> tag optimization (SEM 101)

Much of what constitutes a well-architected webpage is never displayed in the page itself. The contents of the <body> tag are what you see in a browser. But a webpage consists of two major elements, the <body> tag only being one. The content of the <head> tag (and for that matter, the document type declaration (DTD), which precedes the <head> tag in the page’s code, is just as important for search engine optimization ...
Read More

New bot work continues at Bing

Back in December of 2008, we alerted you to the possibility that you might start seeing our new web crawler in your referrer logs. Given the recent excitement around the arrival of Bing, we wanted to take a moment to update you on the latest bot news. As we said earlier, you will see the new Bing crawler user agent string listed as: msnbot/2.0b (+http://search.msn.com/msnbot.htm). Note that we didn’t change the bot’s name for Bing...
Read More