Optimizing your very large site for search — Part 4

Everyone knows great content is fundamental to the success of your site. It is the reason people are looking for your site and it is why they will stay. In the previous three posts from this series, we have discussed how large sites need to expose fewer URLs, how you can help search engines crawl more efficiently, and how not to unintentionally hide your content from crawlers. The common thread here is that we, the search engines, want to have better access to the content that matters on your site.

The reason for this? Content goes a long way in establishing the value of your website for each visitor and for us as a search engine. There is one important truth: search engines like content. For a large site, generating content is not an issue. The issue is ensuring that the most relevant content gets into the index. The solution is creating a content strategy.

Content strategy defined

A content strategy covers the type of content that will best enable you to communicate to your audience and achieve the goals of your site. A content strategy should account for four basic activities:

  1. Identification of content relevant to your site goals
  2. Organization of that content on your site for user accessibility and discoverability
  3. Management of the content as your site grows, including reorganizing and retiring content when necessary
  4. Planning for future site growth with continued additions of new content

A content strategy should help you produce remarkable, attention-grabbing content. Content strategy is a big area, but there are some steps that every large site should be doing to ensure success in the SERP and success with your audience.

Three steps to developing a search friendly content strategy

1. Plan for your audience

It is important to remember that you don’t create sites for search engines—you create sites for people. For many large sites, it can be unclear what the goal or goals are. Knowing the goal you would like to achieve is central to developing a content strategy that works for people, and consequently, for search. Are you trying to sell a product? Educate your customers? Attract readers to breaking news stories?

Begin by assessing your audience and how they will interact with your site:

  • Who uses your content?
  • How do they use your content?
  • When do they use your content?

Formulating what content meets these goals and others can be a great way to reduce unnecessary content and highlight the best content for your audience. This will help ensure that you are exposing the best content to the search engine.

2. Bring together similar content about the same topic in one area

Too many large sites have an old collection of "dangling" content that either lives alone or is lumped together with other "miscellaneous" assets on one page while most of the content related to it is accessible on another page. It’s the classic kitchen junk drawer syndrome. You have nowhere else to put it, so you toss it there. But before you know it, you develop a collection of related content, and you finally create a new page on your site, but you forget to move the junk drawer items to this new place. This all too common situation is typically due to rapid, unmanaged growth of the site’s content collection. Users should be able to find all relevant content easily.

Collect related content in one area or at least make it accessible through one access point on your site. Then regularly reassess your content organization as your content collection grows to prevent the number of these blind alleys and orphaned pieces from getting out of hand. From the user’s perspective, good content is often lost this way. Organization of the content within the pages of your site should make sense to the way users actually think about the content.

The best way to do this is to create a content inventory that helps you understand the potential scope of the content you need to manage and determine how to best organize it. With a lot of available content, the reality is that users may have to visit different subsections of your site to view all of the relevant content they wish to see. In that case, the content organization itself should make it easy for users to understand how different areas are related and categorized.

This practice is common among Information Architects. More companies looking to optimize large sites are making it central to their practice. For more on these practices, read Adam Audette’s article SEO guide to information architecture.

3. Trim around the edges

One thing that surprises me with big sites is how often many URLs lead to outdated content, or worse, a 404 page where content used to exist. Any content strategy should ensure that that all content mentioned or linked to exists. Leading your users and the search crawler to a 404 Page Not Found or an "Under Construction" sign is not a plan for success.

Along with content not found, there really isn’t anything more frustrating than browsing to a site and finding that the page is out of date, a desired product is no longer available, a knowledge base is for an expired version or the news is not current. Lots of content should have an expiration date, and a solid content strategy will review the relevance of pages on a regular basis. That is a good way to ensure that you website provides users with information that is always valid.

Such simple mistakes with content can hurt your credibility with your users and maintains URLs for the crawler to consume that add no value.

To counter this you must create a plan for managing URLs. This plan should include how you will manage reorganizing content and how you will handle the removal of content. A URL plan should include how you plan to:

  • Handle deprecated content
  • Redirect to new versions
  • Deal with error pages

Having a URL plan will help you managing the referrers that link to your content and ensure that your audience can link with confidence to your pages during the months and years ahead. Be sure your plan leverages 301 redirects to reorganized pages to ensure that the URL validity (and thus its search engine value) sticks.

Wrapping it all up

A solid content management strategy is the kind of key deliverable that can keep bringing users to your site and ensure that it is meeting all of your site goals. Consciously and strategically thinking about how your content is put together is a good way to ensure your website is positioned for success. For more information on content strategy, you should read:

Jeremiah Andrick, Product Manager, Live Search Webmaster Center

Join the conversation

  1. Anonymous

    I own a 10-year-old site: http://www.easytourchina.com, but it disappeared a couple of years ago. Though we have done a lot by following the instructions of MSN, it is still not in the live search result. I don’t really know what to do now.

  2. Anonymous

    @China tours

    Just sit it out. I’m convinced Microsoft will pull the plug on Live Search eventually. Google is the only search engine with a clear strategy.

  3. Anonymous

    I diagree with Tinus.  Live Search has made great leaps in the quality of its search engine.

    Once I got all the duplicate content out of the way, I noticed more pages were indexed in Live Search.

  4. Anonymous

    @Tinus/web-directory – aside from relevancy, biggest issue with Live Search is freshness/comprehensiveness. They just take ages to get around large sites.

  5. rickdej

    @all  thanks for the comments.  I appreciate the questions and the concerns.  Remember if you have a specific complaint or problem our forums are a great place to go for more help.

    @tinus I appreciate your frustration, but these posts were not intended to blame anyone, but offer some best practices that are recognized to make a difference for all the search engines.   As for your coverage problem as mentioned in previous posts, coverage like rank is affected by a number of factors.  For instance, your site doesn’t have a lot of links to lower level pages or posts but most links point only to a few pages.  Gathering more high quality links could help improve this.

    Remember that we don’t guarantee 100% coverage for any one site. There are billions of pages on the web and we do our best to crawl and index the best.   We are always looking to improve and I will provide your feedback to our team.


    Jeremiah Andrick

  6. Anonymous

    Good tips :) i really use this for my clients. thanks a lot

  7. Quality Directory

    I have employed the talked about guidelines for developing a content strategy and my inner links appear in Google and Yahoo. I'm still waiting to see how improved Bing is, with regards to that.

  8. luke.sarker

    Really good tips.

  9. Anonymous

    Such good tips for our forum with large index page, hope you could perform better than google

  10. miles2go

    who ever had followed all the four article will understand it better.

  11. Anonymous

    great post,thanks for sharing

  12. modesto

    You can't blame any search engine for not being indexed. I guarantee that if you are not being indexed something is wrong on the site owners end. Just because one search engines indexes a site does not mean another will do so. Every search engine has its' own set of rules, what works for one will not always work for the other. It is the job of the webmaster to figure out the differences. I think one problem is that the search engines simply do not want to be a copycat of another search engine, so some excellent sites may get listed in one search engine but not another. And another thing is that articles like this one and not meant to give all the answers, it is just a guideline, which means that it only contains about 5% of what you need to know in order to be a successful webmaster. You are not going to have most of the answers handed to you on a silver platter, you are going to have to dig and dig and dig until you finally hit a pay day.

  13. Anonymous

    I really like this post.  Thanks for this article,  Anyone got any more info about it? I am now your blog' s rss follower. you are now in my bookmarks.

  14. DanBan

    Nice answers on what to do to get listed into Bing

  15. novintabligh

    I have a publicity site where you can find a lot of different content. How I can organize it in the best way?

Comments are closed.