Get detailed site analysis to solve problems (SEM 101)

Last week I discussed common site errors that can inhibit the search engine crawler (aka bot). I pointed out common areas of concern and recommended a few tools to help diagnose such issues on your site’s pages. This week, I wanted to introduce a new tool for your collection that will be of great service to you in checking and maintaining your site.

The tool, Search Engine Optimization (SEO) Toolkit from the Microsoft Internet Information Server (IIS) team, was released as beta 1 back in June, 2009. I haven’t had a chance to talk about it much in this blog until now. Let’s fix that.

Requirements

The IIS SEO Toolkit runs as an extension of IIS 7. This means you have to have IIS 7 on your local machine (sorry Windows XP users!). Users of Windows 7 are good to go now, and users of Windows Vista can easily install IIS 7 by way of the Web Platform Installer tool, which I’d recommend doing if only to try out this very cool new tool.

Feature set

Once installed, the IIS SEO Toolkit works within your local installation of IIS 7. However, the tool itself is not limited to only doing its SEO analysis on IIS-based websites. No way. It can scan sites running PHP, ASP.NET, and all manner of HTML/XHTML pages.

The IIS SEO Toolkit consists of three separate add-on modules to IIS 7:

Site Analysis. This tool begins with a local, full-featured website crawler engine (aka bot) named IISBot that offers the following features:

  • Configurable number of concurrent requests to allow users to crawl their website without incurring additional processing. This can be configured from 1 to 16 concurrent requests.
  • Support for standard Robots.txt commands, allowing you to specifically dictate which web files and directories IISBot should not crawl.
  • Support for Sitemap files, enabling you to identify important areas of your website to be analyzed.
  • Support for overriding “noindex” and “nofollow” <meta> tag attributes to allow you to analyze pages that are otherwise off-limits to typical search engine bots.
  • Configurable limits for analysis, such as maximum number of URLs to download and maximum number of kilobytes to download per URL.
  • Configurable options for including content from only your directories or the entire site and subdomains.

IISBot examines your website contents, discovering links, downloading the content, and applying a set of validation rules aimed to help you easily identify and troubleshoot common problems. These rules cover a variety of pre-configured site violations, such as broken links, duplicate content, keyword analysis, route analysis, and many more. Other features of the Site Analysis tool include:

  • View detailed summary of Web site analysis results through a rich dashboard.
  • Feature-rich Query Builder interface exposes large amounts of data.
  • Quick access to common tasks.
  • Display detailed information for each URL.
  • View detailed route analysis showing unique routes to better understand the way search engines reach your content.

Using the Site Analysis tool to identify and thus correct violations will help improve the overall quality of your website and thus improve your SEO performance. It also provides a great deal of information about your site’s structure and architecture design, all of which help you maintain the site and eliminate redundancies.

Robots Exclusion Editor. This tool includes a powerful editor to create a new or edit an existing Robots Exclusion Protocol (robots.txt) file. It can leverage the output of a Site Analysis crawl report and allow you to easily add Allow and Disallow file and/or folder entries without having to edit a plain text file, making it less error prone and more reliable. Furthermore, you can run the Site Analysis feature again and see immediately the results of applying your robots.txt file.

Sitemap and Sitemap Index Editor. Similar to the Robots.txt editor, this tool allows you to author Sitemap and Sitemap Index files with the ability to discover both physical and logical (Site Analysis crawler report) view of your site.

Using the tool

Once you have finished the tool installation process, start IIS 7. You’ll find the icons for the IIS SEO Toolkit in their own group called Search Engine Optimization.

We’re going to focus on the Site Analysis tool for this article. When you open that tool, you’ll land on the Site Analysis page, where you can open an existing report or click New Analysis to create a new site report (including updates to existing reports).

When creating a new report, you’ll provide IIS SEO Toolkit with a report name and the site’s root URL to scan (I used a sample site saved on my localhost for these images).

IIS SEO Toolkit crawls the site’s home page and follows the links on the page to all of the other pages on the site and examines the content they contain. Once the site is fully crawled, the tool analyzes the data brought down and creates a summary report on the state of the site.

The site information is categorized in the following groups: Violations, Content, Performance, and Links.

The first report is a list of detected site violations. These violations, prioritized into levels named Errors, Warnings, and Information, are also aggregated into error types that attribute them as violations of either SEO, Standards, Content, or Performance.

Double-clicking on a listing in the Violations Summary tab opens up a new tab containing the Group Details of that violation, where each page containing that violation is identified.

Once you get into the Group Details of a specific violation, you’ll see how the pre-configured rule queries are built. You can then use the query tools to begin your own, detailed exploration of your site’s issues by building and saving your own custom queries.

Double-clicking on a listing there calls up information on that particular violation, including details of the violation and recommended actions to take.

You can change tabs in that dialog box to see the affected page’s header, code content, a listing of inbound and outbound links associated with that page, and analysis of the word usage on that page (which is helpful for correcting missing keyword text, such as image alt attributes, meta descriptions, and page titles).

The Site Analysis Report also provides a list of the pages containing the most violations. Clicking through those listings will also provide you with access to the same kind of deep dive data as mentioned earlier about the affected page.

The IIS SEO Toolkit does more than merely offer violation reports. It takes the data it gathers about your site’s pages and enables you to get aggregated reports on other issues and metrics. For example, the Content tab offers a plethora of site analysis data. You can get a list of all external links, duplicated content, and other very useful information.

The reports also offer information on performance issues on a per page or per content-type basis.

Lastly, you get detailed information about the links on your site.

As you can see, the Site Analysis tool is feature-packed and will be very useful to webmasters and SEOs everywhere. There is not enough time or room in this blog to document everything this tool can do, but I can link to pages that will be of use to you in learning more about it.

Be sure to check out a nice, 14-minute video tutorial introducing the IIS SEO Toolkit, as well as a detailed blog post from IIS team lead Scott Guthrie as he uses it to examine his own personal website. IIS team developer Carlos Aguilar Mares also has a blog in which he regularly does deep-dive explorations into the tool’s extensive feature set. It’s worth following Carlos’ blog for his useful insights, tips, and tricks on getting the most from this great tool.

Keep an eye on the IIS SEO Toolkit’s home page for updates. Remember that it is still in beta, so there’s more to come as the development process continues.

If you have any questions, comments, or suggestions, feel free to post them in our SEM forum. See you again soon…

– Rick DeJarnette, Bing Webmaster Center

***********************************************

UPDATE

We’ve gotten some feedback from webmasters asking how to set up IIS 7 on their computers so they can use the IIS SEO Toolkit. To help those folks out, we’ve published a new blog post to cover that question in detail. Take a look at Getting the IIS SEO Toolkit up and running to learn how to enable IIS 7 on your computer. Thank you all for the helpful feedback!

Rick DeJarnette

Join the conversation

40 comments
  1. Anonymous

    This is a seriously cool tool, Rick. Congrats to you and the team for putting this together; hope to see even more ubiquity (and maybe a web-based version) in the months to come.

  2. Anonymous

    Hello,

    Please code the same tool for Apache ! :)

  3. zhangmin_iichiba

    cool tool , i will install , thanks

  4. zhangmin_iichiba

    My computer is windows XP

    i think this software will update for windows XP user!

  5. smehta_surat

    This is really nice tool..I will surely use this tool.

  6. Quality Directory

    I like Search Engine Optimization Toolkit so much. So informative!

  7. edward.weiss

    Will definetely be using this tool in the near future!

  8. obiz

    really good tool and will use for analysis in future

  9. Bob

    Wow – great detail.  This looks really neat-o.

  10. jackin

    Let me try this amazing tool! The screen shot is really helpful while doing installation!

  11. Anonymous

    That's pretty snazzy. Unfortunately it's still not going to make things as clean as if it were on an apache box, but it's a start!

  12. miles2go

    I am using the tool its great tell you the backlinks, and most important is keyword section. where we check the site status for a keyword.

  13. ghostbb8888

    the backlink and the link scores help me a lot!

  14. wapofibrao

    I canĀ“t believe it has not been developed for XP too … :(

  15. Anonymous

    Any chance of getting a link to more descriptive instructions for using it on Vista?  I downloaded it IIS7 and the Web App installer but am not getting past a blank IIS Manager screen.

    Any guidance is appreciated.

  16. Anonymous

    Never mind!  I found the solution by installing the IIS Manager for Remote Administration tool.  

    :o)

  17. Anonymous

    Why not XP too?

  18. Synergy Connections

    Looks really cool, but unfortunately I am still on XP for the moment.  Are there any similar tools anyone knows about that provide the same functionality?

  19. Dead J. Dona

    okay, no xp? i can't believe…

  20. Anonymous

    It is very nice tool for webmasters all over the world. we can use it and get benefited.

  21. Anonymous

    I have installed this on a Windows 7 machine but I am not getting all of the icons.  Only the Site Analysis is showing up in the IIS Manager.  I ran the install linked about, any ideas?

  22. mitchamazed

    Thanks for the time you spent to make this so clear! Our site, <a href='http://trueshare.com'>TrueShare.com</a&gt; has very good SERPs in Google but gets no results whatsoever in Bing and we had very few in MSN previously. I am hoping that cleaning up some of the issues identified by this great tool will help in the Bing SERPs.  

    I do have a quick question if there is anyone out there reading this that may know… The tool shows trueshare.com and trueshare.com/index.aspx as two different pages when in reality they are serving the same page. Am I losing SEO juice or getting tagged for duplicate content in this situation?

    Thanks!

  23. douggebhardt

    PLEASE PLEASE PLEASE make an XP version of this tool too

  24. MaC69

    Why don't you develop tools that are on the web? Why do you "force" users to install tools with all problems of the different operating systems?

    Your friend Google is much better. You can use every Google webmaster tool with every kind of computer as these tools work on the net!

    Please think about it!

    I still have Xp. I will wait for next upgrades hoping they are for Xp too…or better on the net!

  25. redtubeeu

    very useful tool. Thanks

  26. IndyWebDude

    Finally something completely useful from MS!!!! First really good thing since the old Site Builder's Network!

    Google better be quaking in fear now, MS is back! :-)

  27. brightcreek

    great tool

  28. ataboy

    "Requirements – … IIS 7".

    It would be interesting to include in this blog the percentage of webmasters that do NOT use IIS 7.

  29. Anonymous

    I wish this tool was compatible across different types of servers.

    Zuheb

  30. rickdej

    Once you install the IIS SEO Toolkit on your computer, you can use it to analyze any site on the web, regardless of whether it runs on Apache, IIS, or any other web server platform. You only need IIS 7 on your local computer to run the Toolkit. It works against any website, though. Enjoy!

    Rick

  31. gregorylowe

    I feel really stupid, but where do I go to use the Search Engine Optimization Toolkit?  I followed ALL of your steps above but now I really don't know where to go next. I know it is an extension of IIS 7, but I need to know where to start. Please help.

  32. JteLdz

    Good tool but the installation is so tiresome. You need so many applications installed to get this running :/

  33. ChrisGau

    Newbie question: Toolkit downloaded, now how do I "start" IIS7?

  34. jianlong_lin

    Just downloaded the tool and tried to generate one report. Wow, that is great! This is the tool I was looking for. Thanks!

  35. tru2imag

    Honestly, Google does this, MyAsg does this-more over complication of simple issues that can ither be gathered by tools on line, easily set up under your domain, or most likely even offered by your host.  Why go to this trouble? <a href="http://www.actnowcenter.com">http://www.actnowcenter.com</a&gt; ranks high everywhere but Bing.  As a programmer for 13 years on the web, I thought I could get some insight here.  Too bad.

  36. dealingway

    this site analysis tool is  very good. is very useful To  the promotion my website

Comments are closed.