When we first teamed up with Google and Yahoo to support sitemaps as an industry standard, it was with the express intention of driving the protocol forward.  Making it into a simple, effective way to tell all search engines about the structure of your site.

 

Today, I’m happy to announce the latest addition to the protocol: Sitemap Autodiscovery.  Autodiscovery gives siteowners the ability to easily share their sitemaps with all search engines at once, without the overhead of manually submitting them to each one.  In order to share your sitemap with the crawlers at large, you simply provide the following line in your robots.txt file:

 

                Sitemap: <sitemap_location>

 

Where <sitemap_location> is the complete URL of your Sitemap Index File (or your sitemap file, if you don’t have an index file).  This directive is independent of the user-agent. so as long as it is present in the robots.txt file, all engines will be able to find your sitemaps.  I see this as an important extension to the protocol, as it removes the need to individually submit sitemaps to each engine – simply add one line to your robots.txt file, and we will find your new and updated sitemaps automatically.

 

Although we aren’t ready to start consuming sitemaps quite yet, I encourage you to build a sitemap and add the “sitemap” directive to your robots.txt. As soon as we roll out support (before the end of the year), we will be able to start crawling your sitemap files immediately.  If you want more detail on the sitemaps protocol, including autodiscovery, check out www.sitemaps.org.

 

As always, we are eager to hear your feedback and suggestions.  Please send them our way using this link.

 

Brent Hands, Program Manager, Live Search